Creates a new Evaluation of an MLModel. An MLModel is evaluated on a set of observations associated to a DataSource. Like a DataSource for an MLModel, the DataSource for an Evaluation contains values for the Target Variable. The Evaluation compares the predicted result for each observation to the actual outcome and provides a summary so that you know how effective the MLModel functions on the test data. Evaluation generates a relevant performance metric, such as BinaryAUC, RegressionRMSE or MulticlassAvgFScore based on the corresponding MLModelType: BINARY, REGRESSION or MULTICLASS.
CreateEvaluation is an asynchronous operation. In response to CreateEvaluation, Amazon Machine Learning (Amazon ML) immediately returns and sets the evaluation status to PENDING. After the Evaluation is created and ready for use, Amazon ML sets the status to COMPLETED.
You can use the GetEvaluation operation to check progress of the evaluation during the creation operation.
Parameters:
-
evaluation_id
(EntityId)
–
A user-supplied ID that uniquely identifies the Evaluation.
-
evaluation_name
(Optional[EntityName], default:
None
)
–
A user-supplied name or description of the Evaluation.
-
ml_model_id
(EntityId)
–
The ID of the MLModel to evaluate.
The schema used in creating the MLModel must match the schema of the DataSource used in the Evaluation.
-
evaluation_data_source_id
(EntityId)
–
The ID of the DataSource for the evaluation. The schema of the DataSource must match the schema used to create the MLModel.
Raises:
Source code in docs_src/capo_machine_learning/_services/async_machine_learning.py
| async def create_evaluation(
self,
evaluation_id: "capo_machine_learning.types.entity_id.EntityId",
ml_model_id: "capo_machine_learning.types.entity_id.EntityId",
evaluation_data_source_id: "capo_machine_learning.types.entity_id.EntityId",
*,
config_overrides: Optional[AsyncMachineLearningClientConfig] = None,
evaluation_name: Optional[
"capo_machine_learning.types.entity_name.EntityName"
] = None,
) -> "capo_machine_learning.types.create_evaluation_output.CreateEvaluationOutput":
"""<p>Creates a new <code>Evaluation</code> of an <code>MLModel</code>. An <code>MLModel</code> is evaluated on a set of observations associated to a <code>DataSource</code>. Like a <code>DataSource</code> for an <code>MLModel</code>, the <code>DataSource</code> for an <code>Evaluation</code> contains values for the <code>Target Variable</code>. The <code>Evaluation</code> compares the predicted result for each observation to the actual outcome and provides a summary so that you know how effective the <code>MLModel</code> functions on the test data. Evaluation generates a relevant performance metric, such as BinaryAUC, RegressionRMSE or MulticlassAvgFScore based on the corresponding <code>MLModelType</code>: <code>BINARY</code>, <code>REGRESSION</code> or <code>MULTICLASS</code>. </p> <p> <code>CreateEvaluation</code> is an asynchronous operation. In response to <code>CreateEvaluation</code>, Amazon Machine Learning (Amazon ML) immediately returns and sets the evaluation status to <code>PENDING</code>. After the <code>Evaluation</code> is created and ready for use, Amazon ML sets the status to <code>COMPLETED</code>. </p> <p>You can use the <code>GetEvaluation</code> operation to check progress of the evaluation during the creation operation.</p>
Args:
evaluation_id: <p>A user-supplied ID that uniquely identifies the <code>Evaluation</code>.</p>
evaluation_name: <p>A user-supplied name or description of the <code>Evaluation</code>.</p>
ml_model_id: <p>The ID of the <code>MLModel</code> to evaluate.</p> <p>The schema used in creating the <code>MLModel</code> must match the schema of the <code>DataSource</code> used in the <code>Evaluation</code>.</p>
evaluation_data_source_id: <p>The ID of the <code>DataSource</code> for the evaluation. The schema of the <code>DataSource</code> must match the schema used to create the <code>MLModel</code>.</p>
Raises:
capo_machine_learning.errors.idempotent_parameter_mismatch_exception.IdempotentParameterMismatchException: <p>A second request to use or change an object was not allowed. This can result from retrying a request using a parameter that was not present in the original request.</p>
capo_machine_learning.errors.internal_server_exception.InternalServerException: <p>An error on the server occurred when trying to process a request.</p>
capo_machine_learning.errors.invalid_input_exception.InvalidInputException: <p>An error on the client occurred. Typically, the cause is an invalid input value.</p>
capo_machine_learning.errors.UnknownServiceError: The service returned an error code this client does not model.
"""
async def _handler(
req: "AsyncOperationRequest[capo_machine_learning.types.create_evaluation_input.CreateEvaluationInput]",
) -> AsyncOperationResponse[
"capo_machine_learning.types.create_evaluation_output.CreateEvaluationOutput"
]:
import capo_machine_learning._operations.amazon_ml_20141212.create_evaluation
(
output,
http_response,
) = await capo_machine_learning._operations.amazon_ml_20141212.create_evaluation.async_create_evaluation(
req.options, req.input
)
return AsyncOperationResponse(output=output, response=http_response)
interceptors_, options_ = self.operation_options(
config_overrides, OperationLogging("CreateEvaluation")
)
input_: capo_machine_learning.types.create_evaluation_input.CreateEvaluationInput = {
"evaluation_id": evaluation_id,
"ml_model_id": ml_model_id,
"evaluation_data_source_id": evaluation_data_source_id,
}
if evaluation_name is not None:
input_["evaluation_name"] = evaluation_name
response = await aexecute_pipeline(
AsyncOperationRequest(input=input_, options=options_),
handler=_handler,
interceptors=list(interceptors_),
)
await response.response.aclose()
return response.output
|