invoke_inference_router_async
async
¶
invoke_inference_router_async(
inference_router_name: str,
routing_target_name: str,
input_location: str,
*,
config_overrides: Optional[AsyncSageMakerInferenceRouterRuntimeClientConfig] = None,
content_type: Optional[str] = None,
accept: Optional[str] = None,
custom_attributes: Optional[CustomAttributesHeader] = None,
inference_id: Optional[str] = None,
s3_output_path_extension: Optional[str] = None,
filename: Optional[str] = None,
request_ttl_seconds: Optional[int] = None,
invocation_timeout_seconds: Optional[int] = None,
) -> InvokeInferenceRouterAsyncOutput
Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.
If a destination doesn't start the request within the routing target's MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request before RequestTTLSeconds runs out, or every destination has been tried, the router writes a failure object to the FailureLocation and publishes a notification to the routing target's error topic, if one is configured.
For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.
Parameters:
-
inference_router_name(str) –The name of the inference router to invoke.
-
routing_target_name(str) –The name of the routing target in the inference router. The routing target must be configured for asynchronous inference.
-
content_type(Optional[str], default:None) –The MIME type of the input data in Amazon S3. The router passes it to the destination.
-
accept(Optional[str], default:None) –The MIME type that you want for the inference response. The router passes it to the destination.
-
custom_attributes(Optional[CustomAttributesHeader], default:None) –Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.
-
inference_id(Optional[str], default:None) –An identifier for the inference request. If you don't provide one, SageMaker AI generates one and returns it in the response.
-
input_location(str) –The Amazon S3 URI of the object that contains the input data for the inference. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account. The destination endpoint's execution role reads the object, and the destination's asynchronous inference limits apply to it.
-
s3_output_path_extension(Optional[str], default:None) –The path extension that is appended to the routing target's Amazon S3 output path where the inference response is stored.
-
filename(Optional[str], default:None) –The file name of the inference response stored in Amazon S3. If you don't provide one, SageMaker AI generates one.
-
request_ttl_seconds(Optional[int], default:None) –The time to live of the request, in seconds, measured from when the router accepts it. If no destination starts the request in this time, the request fails. The default is 21,600 seconds (6 hours).
-
invocation_timeout_seconds(Optional[int], default:None) –The maximum time, in seconds, that a destination can spend processing the request before the request is marked as expired. The default is 900 seconds (15 minutes).
Raises:
-
AccessDeniedException–You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
-
InternalFailure–An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
-
RequestTimeoutException–The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
-
ServiceUnavailableException–The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
-
ThrottlingException–The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
-
ValidationException–The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
-
UnknownServiceError–The service returned an error code this client does not model.
Source code in docs_src/capo_sagemaker_inference_router_runtime/_services/async_sage_maker_inference_router_runtime.py
279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 | |