Draft-Guided Parallel Token Generation with Target Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models suffer from high latency, which hinders timely response in certain applications, and existing strategies for reducing latency have limitations.
Innovation Solution
A cooperative system involving a smaller draft language model and a target language model, where the target model verifies draft tokens and produces guidance information in parallel, allowing the draft model to generate tokens in parallel and reduce the number of operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large language model is used to generate tokens, then accuracy is improved, but latency increases
Solution Approach 1:
The system divides the token generation task into two segments: a draft language model that generates candidate tokens quickly, and a target language model that verifies and refines them. This segmentation allows each model to specialize, with the draft model optimizing for speed and the target model for accuracy, thereby reducing overall latency while maintaining high accuracy.
Solution Approach 2:
The draft language model acts as an intermediary between the user query and the final output. It generates initial token candidates that are then passed to the target language model for verification. This intermediary approach allows the system to benefit from both fast generation (draft model) and accurate verification (target model), resolving the latency-accuracy tradeoff.
2Measurement precision
If a large language model is used to verify draft tokens, then accuracy is improved, but computational resources increase
Solution Approach 1:
The target language model does not need to regenerate all tokens from scratch, nor does it need to perform full verification on every draft token. Instead, it performs partial verification only on tokens that need refinement, and uses guidance information to focus computational resources on the most critical verification tasks, thereby reducing overall computational consumption while maintaining accuracy.
Solution Approach 2:
The system changes the operational parameters of the target language model by using guidance information from the draft model to adjust verification focus. This allows the target model to operate more efficiently by concentrating computational resources on tokens with lower confidence scores or higher complexity, rather than uniformly processing all tokens, thus reducing total resource consumption.
3Measurement precision
If tokens are generated auto-regressively one at a time, then accuracy is maintained, but productivity decreases
Solution Approach 1:
The draft language model performs preliminary generation of multiple token candidates in parallel before the target language model verifies them. This preliminary action allows the system to prepare multiple potential outputs simultaneously, then the target model can verify them in a more efficient manner, achieving both high accuracy and improved productivity by avoiding sequential generation.
Solution Approach 2:
The system transitions from one-dimensional sequential generation to two-dimensional parallel processing by generating multiple token candidates simultaneously using the draft model, then verifying them in parallel using the target model. This dimensional change from sequential to parallel processing dramatically increases productivity while maintaining accuracy through the verification step.
Data Source
AI summary
A technique accelerates the generative production of tokens using a target language model that operates in cooperation with a smaller draft language model. In operation, the target language model (1) verifies the accuracy of a first set of draft tokens produced by the draft language model, (2) predicts a new token to follow the last-verified draft token, and (3) generates plural instances of guidance information. The draft language model produces a second set of draft tokens in parallel based on the target output token(s) produced by the target language model and the instances of guidance information. The technique expedites the generation of tokens because the draft language model, due to its size, is able to produce tokens faster than the target language model. The draft language model produces its draft tokens in parallel (at the same time), rather than auto-regressively, which further speeds up token generation.