Self-Speculative Decoding With Forecasted Embeddings for Faster LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative artificial intelligence models, such as large language models, are computationally expensive due to the need for multiple passes through the model for each token generation, which is impractical for devices with limited resources and hinders efficient response generation.
Innovation Solution
Implement self-speculative decoding using a single generative AI model that combines draft and target model functions, with forecasted embeddings to parallelize token generation and verification, reducing computational expense and increasing throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple passes through the large language model are performed for each token generation, then the quality and accuracy of the generated response is improved, but the computational expense and time required increases significantly
Solution Approach 1:
The patent divides the token generation process into two distinct phases: a draft phase using a smaller, faster model to generate multiple candidate tokens, and a verification phase using the larger target model to validate these candidates. This segmentation allows the system to benefit from both the speed of the draft model and the accuracy of the target model, resolving the contradiction between response quality and generation speed.
Solution Approach 2:
The draft model performs preliminary token generation before the target model verification. By pre-generating multiple candidate tokens and filtering them through the draft model first, the system reduces the number of times the computationally expensive target model needs to be invoked, thereby improving token generation speed while maintaining response quality through subsequent verification.
2Measurement precision
If the number of parameters within the large language model is increased to improve response quality, then the computational resource expense increases
Solution Approach 1:
The patent merges the functionalities of two models with different characteristics: a smaller draft model that is energy-efficient and a larger target model that produces high-quality responses. The draft model handles the energetically expensive work of generating multiple candidate tokens, while the target model performs only the verification step, significantly reducing overall computational resource expense while maintaining response quality.
Solution Approach 2:
The draft model creates multiple candidate token copies that are then verified by the target model. Instead of repeatedly invoking the large target model to generate each token, the system uses the draft model to generate candidate copies and only validates them with the target model, reducing the computational burden on the parameter-intensive large model.
3Productivity
If speculative decoding is used to increase token generation speed, then the complexity of the decoding process increases
Solution Approach 1:
The draft model serves as an intermediary between the input prompt and the target model verification process. It generates candidate tokens that mediate the interaction between the user query and the computationally intensive target model, simplifying the overall process by filtering and preparing candidates before they reach the target model, thereby managing complexity while maintaining high generation speed.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for generating a response to a query input in a generative artificial intelligence model. An example method generally includes receiving an input prompt for processing; generating a set of forecasted parameters for the input prompt using a parameter prediction model; generating, using a generative artificial intelligence model, a response to the input prompt based on the input prompt and the set of forecasted parameters; and outputting the generated response.