Skip to content

ModelSpeculativeDecodingConfig

Source code in docs_src/capo_sagemaker/types/model_speculative_decoding_config.py 
12
13
14
15
16
17
18
19
20
class ModelSpeculativeDecodingConfig(TypedDict, closed=True):
    technique: NotRequired[
        "capo_sagemaker.types.model_speculative_decoding_technique.ModelSpeculativeDecodingTechnique"
    ]
    """<p>The speculative decoding technique to apply during model optimization.</p>"""
    training_data_source: NotRequired[
        "capo_sagemaker.types.model_speculative_decoding_training_data_source.ModelSpeculativeDecodingTrainingDataSource"
    ]
    """<p>The location of the training data to use for speculative decoding. The data must be formatted as ShareGPT, OpenAI Completions or OpenAI Chat Completions. The input can also be unencrypted captured data from a SageMaker endpoint as long as the endpoint uses one of the above formats.</p>"""

technique instance-attribute 

technique: NotRequired[ModelSpeculativeDecodingTechnique]

The speculative decoding technique to apply during model optimization.

training_data_source instance-attribute 

training_data_source: NotRequired[ModelSpeculativeDecodingTrainingDataSource]

The location of the training data to use for speculative decoding. The data must be formatted as ShareGPT, OpenAI Completions or OpenAI Chat Completions. The input can also be unencrypted captured data from a SageMaker endpoint as long as the endpoint uses one of the above formats.