StoppingCondition
¶
Used by
Source code in docs_src/capo_sagemaker/types/stopping_condition.py
max_pending_time_in_seconds
instance-attribute
¶
max_pending_time_in_seconds: NotRequired[MaxPendingTimeInSeconds]
The maximum length of time, in seconds, that a training or compilation job can be pending before it is stopped.
When working with training jobs that use capacity from training plans, not all Pending job states count against the MaxPendingTimeInSeconds limit. The following scenarios do not increment the MaxPendingTimeInSeconds counter:
-
The plan is in a
Scheduledstate: Jobs queued (inPendingstatus) before a plan's start date (waiting for scheduled start time) -
Between capacity reservations: Jobs temporarily back to
Pendingstatus between two capacity reservation periods
MaxPendingTimeInSeconds only increments when jobs are actively waiting for capacity in an Active plan.
-
MaxPendingTimeInSecondstakes effect only for jobs that request accelerated computing instance types, such as instances in theml.p,ml.g, andml.trnfamilies. It has no effect on jobs that request CPU-only instance types. -
If the job specifies
InstancePreferences,MaxPendingTimeInSecondsbounds the total time SageMaker spends working through your list of instance types. It is not applied per instance type preference, and takes effect only when the list includes at least one accelerated computing instance type.
max_runtime_in_seconds
instance-attribute
¶
max_runtime_in_seconds: NotRequired[MaxRuntimeInSeconds]
The maximum length of time, in seconds, that a training or compilation job can run before it is stopped.
For compilation jobs, if the job does not complete during this time, a TimeOut error is generated. We recommend starting with 900 seconds and increasing as necessary based on your model.
For all other jobs, if the job does not complete during this time, SageMaker ends the job. When RetryStrategy is specified in the job request, MaxRuntimeInSeconds specifies the maximum time for all of the attempts in total, not each individual attempt. The default value is 1 day. The maximum value is 28 days.
The maximum time that a TrainingJob can run in total, including any time spent publishing metrics or archiving and uploading models after it has been stopped, is 30 days.
max_wait_time_in_seconds
instance-attribute
¶
max_wait_time_in_seconds: NotRequired[MaxWaitTimeInSeconds]
The maximum length of time, in seconds, that a managed Spot training job has to complete. It is the amount of time spent waiting for Spot capacity plus the amount of time the job can run. It must be equal to or greater than MaxRuntimeInSeconds. If the job does not complete during this time, SageMaker ends the job.
When RetryStrategy is specified in the job request, MaxWaitTimeInSeconds specifies the maximum time for all of the attempts in total, not each individual attempt.