Speculative Decoding Engines With Adaptive Context Length
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges with slow inference due to autoregressive decoding, leading to undesirably slow user experiences and high memory requirements, especially on resource-constrained devices, and are prone to hallucinations, which can result in inaccurate responses.
Innovation Solution
Implementing a speculative decoding system that uses smaller draft models to generate tokens in parallel, which are verified by a larger target model, optimizing context length based on error rates to balance throughput and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If autoregressive decoding is used in large language models, then the model can generate coherent and contextually accurate responses, but the inference speed becomes slow and memory requirements increase
Solution Approach 1:
The draft model generates candidate tokens in advance before the target model verification step. This preliminary action allows the system to prepare multiple potential outputs simultaneously, reducing the sequential processing bottleneck and improving overall inference speed while maintaining accuracy through subsequent verification.
Solution Approach 2:
The decoding process is segmented into two distinct phases: draft model generation and target model verification. This segmentation allows parallel processing of multiple candidate tokens during the draft phase, followed by efficient verification in the target phase, thereby improving productivity without sacrificing the reliability ensured by the target model.
2Reliability
If context length is increased to reduce error rate, then accuracy improves, but computational resources and memory consumption increase
Solution Approach 1:
The draft model uses a reduced context length (partial action) to generate candidate tokens quickly with lower resource consumption. While this may produce some erroneous candidates, the subsequent target model verification step corrects these errors, achieving acceptable accuracy without the full resource cost of using complete context length throughout the entire decoding process.
Data Source
AI summary
The present disclosure provides systems and methods for: obtaining a request; obtaining, from a draft model, a first plurality of draft tokens based on the request; determining an error rate associated with the first plurality of draft tokens; determining a modified context length for the draft model based on the error rate; and configuring the draft model based on the modified context length.