Speculative Decoding with Draft-Model Verification for LLM Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative artificial intelligence models, such as large language models, are computationally expensive due to the need for multiple passes through the model for each token generation, which is impractical for devices with limited resources and can strain memory bandwidth.
Innovation Solution
Implementing a speculative decoding technique using a smaller draft model to generate tokens in sequence, followed by a target model for verification, with recursive rejection sampling and a constant beam width to optimize token generation and reduce computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a pass through a large language model is used to generate each token in a response, then the accuracy and quality of the generated response is maintained, but the computational expense and processing time increase significantly
Solution Approach 1:
The draft model performs preliminary token generation in advance, creating a set of candidate tokens that are then verified by the target model. This preliminary action allows the system to generate multiple token candidates simultaneously rather than sequentially, improving throughput while maintaining accuracy through subsequent verification.
Solution Approach 2:
The token generation process is segmented into two distinct stages: (1) draft model generates candidate tokens without full verification, and (2) target model verifies the generated tokens. This segmentation allows parallel processing of multiple token candidates in the first stage, increasing overall productivity while the second stage ensures accuracy.
2Measurement precision
If the number of parameters within the large language model increases to improve response quality, then the model's accuracy improves, but the computational resource expense and memory bandwidth requirements increase
Solution Approach 1:
The draft model serves as an intermediary between the input prompt and the target model. It generates candidate tokens that are then verified by the target model, reducing the direct computational burden on the larger target model while maintaining response quality through the verification process.
Solution Approach 2:
The draft model creates a simplified copy or approximation of the token generation process, generating candidate tokens that are then verified by the target model. This copying approach allows parallel generation of multiple candidates with reduced computational resources, while the target model verification ensures quality.
3Measurement precision
If multiple passes through the large language model are performed to generate each token, then the probability distribution accuracy is maintained, but the memory bandwidth strain and processing time increase
Solution Approach 1:
The draft model performs preliminary token generation in parallel, creating multiple candidate tokens with associated probability distributions. This preliminary action reduces the need for multiple sequential passes through the target model, maintaining probability distribution accuracy while reducing processing time.
Solution Approach 2:
The system maintains continuous useful action by having the draft model continuously generate candidate tokens while the target model verifies them in parallel. This continuous parallel processing maintains probability distribution accuracy without the delays associated with sequential multiple passes.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for efficiently generating a response to a query input in a generative artificial intelligence model. An example method generally includes generating, based on an input prompt and using a first machine learning model, a set of tokens including one or more subsets of tokens. Each respective subset of the one or more subsets corresponds to a respective portion of a response to the input prompt and includes a fixed number of tokens corresponding to a beam width for a beam search through the set of tokens. The set of tokens is output to a second machine learning model for verification, and information identifying a selected sequence of tokens from the generated set of tokens is received from the second machine learning model. The selected sequence of tokens is output as the response to the input prompt.