Skip to content

InferenceConfiguration

Source code in docs_src/capo_bedrock_agentcore_control/types/inference_configuration.py 
11
12
13
14
15
16
17
18
19
20
21
class InferenceConfiguration(TypedDict, closed=True):
    max_tokens: NotRequired["int"]
    """<p> The maximum number of tokens to generate in the model response during evaluation. </p>"""
    temperature: NotRequired["float"]
    """<p> The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs. </p>"""
    top_p: NotRequired["float"]
    """<p> The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices. </p>"""
    stop_sequences: NotRequired[
        "capo_bedrock_agentcore_control.types.non_empty_string_list.NonEmptyStringList"
    ]
    """<p> The list of sequences that will cause the model to stop generating tokens when encountered. </p>"""

max_tokens instance-attribute 

max_tokens: NotRequired[int]

The maximum number of tokens to generate in the model response during evaluation.

stop_sequences instance-attribute 

stop_sequences: NotRequired[NonEmptyStringList]

The list of sequences that will cause the model to stop generating tokens when encountered.

temperature instance-attribute 

temperature: NotRequired[float]

The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.

top_p instance-attribute 

top_p: NotRequired[float]

The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.