Reboots specific nodes within a SageMaker HyperPod cluster using a soft recovery mechanism. BatchRebootClusterNodes performs a graceful reboot of the specified nodes by calling the Amazon Elastic Compute Cloud RebootInstances API, which attempts to cleanly shut down the operating system before restarting the instance.
This operation is useful for recovering from transient issues or applying certain configuration changes that require a restart.
-
Rebooting a node may cause temporary service interruption for workloads running on that node. Ensure your workloads can handle node restarts or use appropriate scheduling to minimize impact.
-
You can reboot up to 25 nodes in a single request.
-
For SageMaker HyperPod clusters using the Slurm workload manager, ensure rebooting nodes will not disrupt critical cluster operations.
Parameters:
-
cluster_name
(Optional[ClusterNameOrArn], default:
None
)
–
The name or Amazon Resource Name (ARN) of the SageMaker HyperPod cluster containing the nodes to reboot.
-
node_ids
(Optional[ClusterNodeIds], default:
None
)
–
A list of EC2 instance IDs to reboot using soft recovery. You can specify between 1 and 25 instance IDs.
-
Either NodeIds or NodeLogicalIds must be provided (or both), but at least one is required.
-
Each instance ID must follow the pattern i- followed by 17 hexadecimal characters (for example, i-0123456789abcdef0).
-
node_logical_ids
(Optional[ClusterNodeLogicalIdList], default:
None
)
–
A list of logical node IDs to reboot using soft recovery. You can specify between 1 and 25 logical node IDs.
The NodeLogicalId is a unique identifier that persists throughout the node's lifecycle and can be used to track nodes that are still being provisioned and don't yet have an EC2 instance ID assigned.
-
This parameter is only supported for clusters using Continuous as the NodeProvisioningMode. For clusters using the default provisioning mode, use NodeIds instead.
-
Either NodeIds or NodeLogicalIds must be provided (or both), but at least one is required.
Raises:
Source code in docs_src/capo_sagemaker/_services/async_sage_maker.py
| async def batch_reboot_cluster_nodes(
self,
*,
config_overrides: Optional[AsyncSageMakerClientConfig] = None,
cluster_name: Optional[
"capo_sagemaker.types.cluster_name_or_arn.ClusterNameOrArn"
] = None,
node_ids: Optional[
"capo_sagemaker.types.cluster_node_ids.ClusterNodeIds"
] = None,
node_logical_ids: Optional[
"capo_sagemaker.types.cluster_node_logical_id_list.ClusterNodeLogicalIdList"
] = None,
) -> "capo_sagemaker.types.batch_reboot_cluster_nodes_response.BatchRebootClusterNodesResponse":
"""<p>Reboots specific nodes within a SageMaker HyperPod cluster using a soft recovery mechanism. <code>BatchRebootClusterNodes</code> performs a graceful reboot of the specified nodes by calling the Amazon Elastic Compute Cloud <code>RebootInstances</code> API, which attempts to cleanly shut down the operating system before restarting the instance.</p> <p>This operation is useful for recovering from transient issues or applying certain configuration changes that require a restart.</p> <note> <ul> <li> <p>Rebooting a node may cause temporary service interruption for workloads running on that node. Ensure your workloads can handle node restarts or use appropriate scheduling to minimize impact.</p> </li> <li> <p>You can reboot up to 25 nodes in a single request.</p> </li> <li> <p>For SageMaker HyperPod clusters using the Slurm workload manager, ensure rebooting nodes will not disrupt critical cluster operations.</p> </li> </ul> </note>
Args:
cluster_name: <p>The name or Amazon Resource Name (ARN) of the SageMaker HyperPod cluster containing the nodes to reboot.</p>
node_ids: <p>A list of EC2 instance IDs to reboot using soft recovery. You can specify between 1 and 25 instance IDs.</p> <note> <ul> <li> <p>Either <code>NodeIds</code> or <code>NodeLogicalIds</code> must be provided (or both), but at least one is required.</p> </li> <li> <p>Each instance ID must follow the pattern <code>i-</code> followed by 17 hexadecimal characters (for example, <code>i-0123456789abcdef0</code>).</p> </li> </ul> </note>
node_logical_ids: <p>A list of logical node IDs to reboot using soft recovery. You can specify between 1 and 25 logical node IDs.</p> <p>The <code>NodeLogicalId</code> is a unique identifier that persists throughout the node's lifecycle and can be used to track nodes that are still being provisioned and don't yet have an EC2 instance ID assigned.</p> <important> <ul> <li> <p>This parameter is only supported for clusters using <code>Continuous</code> as the <code>NodeProvisioningMode</code>. For clusters using the default provisioning mode, use <code>NodeIds</code> instead.</p> </li> <li> <p>Either <code>NodeIds</code> or <code>NodeLogicalIds</code> must be provided (or both), but at least one is required.</p> </li> </ul> </important>
Raises:
capo_sagemaker.errors.resource_not_found.ResourceNotFound: <p>Resource being access is not found.</p>
capo_sagemaker.errors.UnknownServiceError: The service returned an error code this client does not model.
"""
async def _handler(
req: "AsyncOperationRequest[capo_sagemaker.types.batch_reboot_cluster_nodes_request.BatchRebootClusterNodesRequest]",
) -> AsyncOperationResponse[
"capo_sagemaker.types.batch_reboot_cluster_nodes_response.BatchRebootClusterNodesResponse"
]:
import capo_sagemaker._operations.sage_maker.batch_reboot_cluster_nodes
(
output,
http_response,
) = await capo_sagemaker._operations.sage_maker.batch_reboot_cluster_nodes.async_batch_reboot_cluster_nodes(
req.options, req.input
)
return AsyncOperationResponse(output=output, response=http_response)
interceptors_, options_ = self.operation_options(
config_overrides, OperationLogging("BatchRebootClusterNodes")
)
input_: capo_sagemaker.types.batch_reboot_cluster_nodes_request.BatchRebootClusterNodesRequest = {}
if cluster_name is not None:
input_["cluster_name"] = cluster_name
if node_ids is not None:
input_["node_ids"] = node_ids
if node_logical_ids is not None:
input_["node_logical_ids"] = node_logical_ids
response = await aexecute_pipeline(
AsyncOperationRequest(input=input_, options=options_),
handler=_handler,
interceptors=list(interceptors_),
)
await response.response.aclose()
return response.output
|