invoke_inference_router
async
¶
invoke_inference_router(
inference_router_name: str,
routing_target_name: str,
body: InferenceRouterBodyBlob,
*,
config_overrides: Optional[AsyncSageMakerInferenceRouterRuntimeClientConfig] = None,
content_type: Optional[str] = None,
accept: Optional[str] = None,
inference_id: Optional[str] = None,
prefix_aware_id: Optional[str] = None,
custom_attributes: Optional[CustomAttributesHeader] = None,
) -> InvokeInferenceRouterOutput
Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
-
The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.
-
Your model has up to 60 seconds for each attempt, the same limit as
InvokeEndpoint. If your account has a higher model execution timeout forInvokeEndpoint, it doesn't apply through an inference router. For models that take longer, useInvokeInferenceRouterAsync.
If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.
Parameters:
-
inference_router_name(str) –The name of the inference router to invoke.
-
routing_target_name(str) –The name of the routing target in the inference router. The routing target lists the destinations that can serve the request.
-
content_type(Optional[str], default:None) –The MIME type of the input data in the request body. The router passes it to the destination.
-
accept(Optional[str], default:None) –The MIME type that you want for the inference response. The router passes it to the destination.
-
inference_id(Optional[str], default:None) –An identifier for the inference request. The router passes it to the destination.
-
prefix_aware_id(Optional[str], default:None) –An optional, stable identifier that serves as a routing hint for prefix-aware routing. The router passes it to the destination that serves the request. Within that destination, requests with the same prefix and the same identifier go to the same instance. If requests from different applications might have the same prompt prefix, set a different identifier for each application to differentiate their routing decisions.
Applies only to destination endpoints configured with a
RoutingStrategyofPREFIX_AWARE. -
custom_attributes(Optional[CustomAttributesHeader], default:None) –Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.
-
body(InferenceRouterBodyBlob) –The input data for the inference, in the format that your model expects. The body can't be empty.
Raises:
-
AccessDeniedException–You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
-
InternalDependencyException–The request failed because of an error in an internal dependency. Retry the request with exponential backoff.
-
InternalFailure–An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
-
ModelError–The model in the destination that served the request returned an error. Check
OriginalStatusCode,OriginalMessage, and the destination's logs in Amazon CloudWatch. -
ModelNotReadyException–The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.
-
RequestTimeoutException–The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
-
ServiceUnavailableException–The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
-
ThrottlingException–The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
-
ValidationException–The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
-
UnknownServiceError–The service returned an error code this client does not model.
Examples:
Invoke an inference router The following example sends a JSON request to the my-target routing target of the my-router inference router. The response shows that a destination in the router's Region served the request.
>>> await client.invoke_inference_router(inference_router_name='my-router', routing_target_name='my-target', content_type='application/json', body='{"instances": [[1.0, 2.0, 5.0]]}')
Source code in docs_src/capo_sagemaker_inference_router_runtime/_services/async_sage_maker_inference_router_runtime.py
166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 | |