Skip to content

invoke_inference_router async

invoke_inference_router(
    inference_router_name: str,
    routing_target_name: str,
    body: InferenceRouterBodyBlob,
    *,
    config_overrides: Optional[AsyncSageMakerInferenceRouterRuntimeClientConfig] = None,
    content_type: Optional[str] = None,
    accept: Optional[str] = None,
    inference_id: Optional[str] = None,
    prefix_aware_id: Optional[str] = None,
    custom_attributes: Optional[CustomAttributesHeader] = None,
) -> InvokeInferenceRouterOutput

Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.

The following limits apply:

  • The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.

  • Your model has up to 60 seconds for each attempt, the same limit as InvokeEndpoint. If your account has a higher model execution timeout for InvokeEndpoint, it doesn't apply through an inference router. For models that take longer, use InvokeInferenceRouterAsync.

If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.

The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.

Parameters:

  • inference_router_name (str) –

    The name of the inference router to invoke.

  • routing_target_name (str) –

    The name of the routing target in the inference router. The routing target lists the destinations that can serve the request.

  • content_type (Optional[str], default: None ) –

    The MIME type of the input data in the request body. The router passes it to the destination.

  • accept (Optional[str], default: None ) –

    The MIME type that you want for the inference response. The router passes it to the destination.

  • inference_id (Optional[str], default: None ) –

    An identifier for the inference request. The router passes it to the destination.

  • prefix_aware_id (Optional[str], default: None ) –

    An optional, stable identifier that serves as a routing hint for prefix-aware routing. The router passes it to the destination that serves the request. Within that destination, requests with the same prefix and the same identifier go to the same instance. If requests from different applications might have the same prompt prefix, set a different identifier for each application to differentiate their routing decisions.

    Applies only to destination endpoints configured with a RoutingStrategy of PREFIX_AWARE.

  • custom_attributes (Optional[CustomAttributesHeader], default: None ) –

    Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.

  • body (InferenceRouterBodyBlob) –

    The input data for the inference, in the format that your model expects. The body can't be empty.

Raises:

  • AccessDeniedException –

    You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.

  • InternalDependencyException –

    The request failed because of an error in an internal dependency. Retry the request with exponential backoff.

  • InternalFailure –

    An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.

  • ModelError –

    The model in the destination that served the request returned an error. Check OriginalStatusCode, OriginalMessage, and the destination's logs in Amazon CloudWatch.

  • ModelNotReadyException –

    The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.

  • RequestTimeoutException –

    The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.

  • ServiceUnavailableException –

    The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.

  • ThrottlingException –

    The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.

  • ValidationException –

    The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.

  • UnknownServiceError –

    The service returned an error code this client does not model.

Examples:

Invoke an inference router The following example sends a JSON request to the my-target routing target of the my-router inference router. The response shows that a destination in the router's Region served the request.

>>> await client.invoke_inference_router(inference_router_name='my-router', routing_target_name='my-target', content_type='application/json', body='{"instances": [[1.0, 2.0, 5.0]]}')
Source code in docs_src/capo_sagemaker_inference_router_runtime/_services/async_sage_maker_inference_router_runtime.py
async def invoke_inference_router(
    self,
    inference_router_name: str,
    routing_target_name: str,
    body: "capo_sagemaker_inference_router_runtime.types.inference_router_body_blob.InferenceRouterBodyBlob",
    *,
    config_overrides: Optional[
        AsyncSageMakerInferenceRouterRuntimeClientConfig
    ] = None,
    content_type: Optional[str] = None,
    accept: Optional[str] = None,
    inference_id: Optional[str] = None,
    prefix_aware_id: Optional[str] = None,
    custom_attributes: Optional[
        "capo_sagemaker_inference_router_runtime.types.custom_attributes_header.CustomAttributesHeader"
    ] = None,
) -> "capo_sagemaker_inference_router_runtime.types.invoke_inference_router_output.InvokeInferenceRouterOutput":
    """<p>Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.</p> <p>The following limits apply:</p> <ul> <li> <p>The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.</p> </li> <li> <p>Your model has up to 60 seconds for each attempt, the same limit as <code>InvokeEndpoint</code>. If your account has a higher model execution timeout for <code>InvokeEndpoint</code>, it doesn't apply through an inference router. For models that take longer, use <code>InvokeInferenceRouterAsync</code>.</p> </li> </ul> <p>If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.</p> <p>The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/inference-router.html">Route requests across Regions with inference routers</a>.</p>

    Args:
        inference_router_name: <p>The name of the inference router to invoke.</p>
        routing_target_name: <p>The name of the routing target in the inference router. The routing target lists the destinations that can serve the request.</p>
        content_type: <p>The MIME type of the input data in the request body. The router passes it to the destination.</p>
        accept: <p>The MIME type that you want for the inference response. The router passes it to the destination.</p>
        inference_id: <p>An identifier for the inference request. The router passes it to the destination.</p>
        prefix_aware_id: <p>An optional, stable identifier that serves as a routing hint for prefix-aware routing. The router passes it to the destination that serves the request. Within that destination, requests with the same prefix and the same identifier go to the same instance. If requests from different applications might have the same prompt prefix, set a different identifier for each application to differentiate their routing decisions.</p> <p>Applies only to destination endpoints configured with a <code>RoutingStrategy</code> of <code>PREFIX_AWARE</code>.</p>
        custom_attributes: <p>Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.</p>
        body: <p>The input data for the inference, in the format that your model expects. The body can't be empty.</p>

    Raises:
        capo_sagemaker_inference_router_runtime.errors.access_denied_exception.AccessDeniedException: <p>You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.</p>
        capo_sagemaker_inference_router_runtime.errors.internal_dependency_exception.InternalDependencyException: <p>The request failed because of an error in an internal dependency. Retry the request with exponential backoff.</p>
        capo_sagemaker_inference_router_runtime.errors.internal_failure.InternalFailure: <p>An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.</p>
        capo_sagemaker_inference_router_runtime.errors.model_error.ModelError: <p>The model in the destination that served the request returned an error. Check <code>OriginalStatusCode</code>, <code>OriginalMessage</code>, and the destination's logs in Amazon CloudWatch.</p>
        capo_sagemaker_inference_router_runtime.errors.model_not_ready_exception.ModelNotReadyException: <p>The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.</p>
        capo_sagemaker_inference_router_runtime.errors.request_timeout_exception.RequestTimeoutException: <p>The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.</p>
        capo_sagemaker_inference_router_runtime.errors.service_unavailable_exception.ServiceUnavailableException: <p>The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.</p>
        capo_sagemaker_inference_router_runtime.errors.throttling_exception.ThrottlingException: <p>The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.</p>
        capo_sagemaker_inference_router_runtime.errors.validation_exception.ValidationException: <p>The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.</p>
        capo_sagemaker_inference_router_runtime.errors.UnknownServiceError: The service returned an error code this client does not model.

    Examples:
        Invoke an inference router
        The following example sends a JSON request to the my-target routing target of the my-router inference router. The response shows that a destination in the router's Region served the request.

        >>> await client.invoke_inference_router(inference_router_name='my-router', routing_target_name='my-target', content_type='application/json', body='{"instances": [[1.0, 2.0, 5.0]]}')
    """

    async def _handler(
        req: "AsyncOperationRequest[capo_sagemaker_inference_router_runtime.types.invoke_inference_router_input.InvokeInferenceRouterInput]",
    ) -> AsyncOperationResponse[
        "capo_sagemaker_inference_router_runtime.types.invoke_inference_router_output.InvokeInferenceRouterOutput"
    ]:
        import capo_sagemaker_inference_router_runtime._operations.sage_maker_inference_router_runtime.invoke_inference_router

        (
            output,
            http_response,
        ) = await capo_sagemaker_inference_router_runtime._operations.sage_maker_inference_router_runtime.invoke_inference_router.async_invoke_inference_router(
            req.options, req.input
        )
        return AsyncOperationResponse(output=output, response=http_response)

    def _redact_input(
        input_: "capo_sagemaker_inference_router_runtime.types.invoke_inference_router_input.InvokeInferenceRouterInput",
    ) -> dict[str, Any]:
        import capo_sagemaker_inference_router_runtime.types.invoke_inference_router_input

        return capo_sagemaker_inference_router_runtime.types.invoke_inference_router_input.redact_sensitive(
            input_
        )

    def _redact_output(
        output: "capo_sagemaker_inference_router_runtime.types.invoke_inference_router_output.InvokeInferenceRouterOutput",
    ) -> dict[str, Any]:
        import capo_sagemaker_inference_router_runtime.types.invoke_inference_router_output

        return capo_sagemaker_inference_router_runtime.types.invoke_inference_router_output.redact_sensitive(
            output
        )

    interceptors_, options_ = self.operation_options(
        config_overrides,
        OperationLogging(
            "InvokeInferenceRouter",
            redact_input=_redact_input,
            redact_output=_redact_output,
        ),
    )
    input_: capo_sagemaker_inference_router_runtime.types.invoke_inference_router_input.InvokeInferenceRouterInput = {
        "inference_router_name": inference_router_name,
        "routing_target_name": routing_target_name,
        "body": body,
    }
    if content_type is not None:
        input_["content_type"] = content_type
    if accept is not None:
        input_["accept"] = accept
    if inference_id is not None:
        input_["inference_id"] = inference_id
    if prefix_aware_id is not None:
        input_["prefix_aware_id"] = prefix_aware_id
    if custom_attributes is not None:
        input_["custom_attributes"] = custom_attributes

    response = await aexecute_pipeline(
        AsyncOperationRequest(input=input_, options=options_),
        handler=_handler,
        interceptors=list(interceptors_),
    )
    await response.response.aclose()
    return response.output