Skip to content

InferenceConfiguration

Source code in docs_src/capo_bedrock/types/inference_configuration.py 
11
12
13
14
15
16
17
18
19
20
21
class InferenceConfiguration(TypedDict, closed=True):
    max_tokens: NotRequired["int"]
    """<p>The maximum number of tokens to allow in the generated response. The default value is the maximum allowed value for the model that you are using.</p>"""
    temperature: NotRequired["float"]
    """<p>The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options, while a higher value makes the model more likely to choose lower-probability options.</p>"""
    top_p: NotRequired["float"]
    """<p>The percentage of most-likely candidates that the model considers for the next token. For example, if you choose a value of 0.8 for <code>topP</code>, the model selects from the top 80% of the probability distribution of tokens that could be next in the sequence.</p>"""
    stop_sequences: NotRequired[
        "capo_bedrock.types.non_empty_string_list.NonEmptyStringList"
    ]
    """<p>A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response.</p>"""

max_tokens instance-attribute 

max_tokens: NotRequired[int]

The maximum number of tokens to allow in the generated response. The default value is the maximum allowed value for the model that you are using.

stop_sequences instance-attribute 

stop_sequences: NotRequired[NonEmptyStringList]

A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response.

temperature instance-attribute 

temperature: NotRequired[float]

The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options, while a higher value makes the model more likely to choose lower-probability options.

top_p instance-attribute 

top_p: NotRequired[float]

The percentage of most-likely candidates that the model considers for the next token. For example, if you choose a value of 0.8 for topP, the model selects from the top 80% of the probability distribution of tokens that could be next in the sequence.