Speculative Decoding With Draft-Model Verification for Faster Token Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative artificial intelligence models are computationally expensive due to the need for multiple passes through large language models to generate responses, which is impractical for devices with limited resources and can hinder other tasks by consuming memory bandwidth.
Innovation Solution
Implementing a smaller draft model to speculatively generate tokens, which are then verified by a target model using recursive rejection sampling, allowing for efficient generation and verification of token sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large language model is used to generate each token sequentially, then the response generation is accurate and comprehensive, but the computational expense increases significantly
Solution Approach 1:
The patent divides the token generation process into two segments: a draft model that generates multiple candidate tokens in parallel, and a target model that verifies and selects from these candidates. This segmentation allows the computationally intensive verification step to be performed only on a subset of candidates rather than generating each token sequentially through a large model, thereby reducing overall computational expense while maintaining accuracy.
Solution Approach 2:
The patent uses a draft model that is a simplified copy or approximation of the target model. This draft model generates multiple candidate tokens quickly, and then the target model verifies these candidates. The draft model acts as a computational shortcut that reduces the burden on the full target model, effectively copying its functionality at a lower computational cost for the generation phase.
2Productivity
If multiple passes through the large language model are performed for token generation, then the response is generated comprehensively, but the memory bandwidth consumption increases
Solution Approach 1:
The draft model performs preliminary generation of multiple candidate tokens before the target model processes them. This preliminary action creates a pool of candidates that the target model can then verify in parallel, eliminating the need for sequential passes through the large model for each token. The preliminary generation handles the bulk of the computational work, reducing memory bandwidth consumption during the verification phase.
Solution Approach 2:
The patent transitions from a sequential, one-dimensional token generation approach to a parallel, multi-dimensional approach where multiple candidate tokens are generated and verified simultaneously. By processing multiple tokens in parallel across different computational dimensions, the system maintains comprehensive response generation while significantly reducing memory bandwidth consumption compared to sequential processing.
3Use of energy by moving object
If a smaller draft model is used for speculative generation, then the computational expense is reduced, but the probability distribution accuracy may be compromised
Solution Approach 1:
The draft model serves as an intermediary between the input prompt and the final output. It generates candidate tokens that are then verified by the target model, which adjusts the probability distribution based on the verified candidates. This intermediary approach allows the draft model to perform computation at lower cost while the target model ensures accuracy through verification and distribution adjustment, resolving the trade-off between computational expense and accuracy.
Solution Approach 2:
The target model provides feedback to the draft model by verifying candidates and adjusting the probability distribution. This feedback mechanism ensures that the draft model's speculative generation is guided by the target model's accuracy requirements, allowing the system to maintain probability distribution accuracy while benefiting from the draft model's lower computational cost. The feedback loop continuously refines the generation process.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for generating a response to a query input in a generative artificial intelligence model. An example method generally includes receiving a plurality of sets of tokens generated based on an input prompt and a first generative artificial intelligence model, each set of tokens in the plurality of sets of tokens corresponding to a candidate response to the input prompt; selecting, using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens, a set of tokens from the plurality of sets of tokens; and outputting the selected set of tokens as a response to the input prompt.