Self-Speculative AI Decoding With Layer-Subset Token Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative AI models face challenges with high computational overhead, latency, and memory demands during inference, particularly in resource-constrained environments, and existing speculative decoding methods often require additional training or complex system integration, leading to diminished efficacy.
Innovation Solution
The method employs skippy simultaneous self-speculative decoding (S3D) that utilizes a subset of a target model's layers to generate candidate tokens, which are then verified by the full model, reducing latency and memory requirements while maintaining output quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If autoregressive decoding is used to generate tokens sequentially, then output quality is maintained, but inference latency increases substantially
Solution Approach 1:
The draft model generates candidate tokens in advance before the target model verifies them. By performing preliminary token generation using a lighter draft model, the system reduces the number of sequential steps the heavy target model must execute, thereby lowering inference latency while maintaining output quality through subsequent verification
Solution Approach 2:
The decoding process is segmented into two distinct phases: draft token generation by the draft model and verification by the target model. This segmentation allows the computationally intensive verification to be performed only on candidate tokens rather than all possible tokens, reducing overall inference time while preserving output quality
2Productivity
If a separate smaller AI model is used for speculative decoding, then inference speed improves, but memory overhead and system complexity increase
Solution Approach 1:
The draft model is designed to be a lightweight, universal version of the target model that can generate candidate tokens for various input sequences. By using the same model architecture type but with fewer parameters, it provides multi-functional capability for speculative decoding without requiring entirely different system components, thus improving inference speed while limiting complexity increases
Solution Approach 2:
Instead of creating a completely new specialized model, the system creates a simplified copy of the target model architecture with reduced parameters. This copied draft model inherits the general language generation capabilities while being computationally lighter, enabling speculative decoding to improve inference speed without proportionally increasing system complexity
3Device complexity
If model compression techniques are applied, then computational complexity is reduced, but output fidelity deteriorates
Solution Approach 1:
The draft model acts as an intermediary between the input sequence and the target model. It generates candidate tokens that are then verified by the full-capability target model. This intermediary approach allows the system to use a computationally simple draft model while maintaining high output fidelity through the target model's verification, thus reducing computational complexity without sacrificing output quality
Solution Approach 2:
The draft model performs preliminary filtering of candidate tokens before they reach the target model. By pre-generating only the most promising candidate tokens based on simplified computations, the system reduces the computational burden on the target model while ensuring that high-fidelity output is maintained through selective verification of the most likely candidates
Data Source
AI summary
A method for controlling an artificial intelligence (AI) device can include receiving, by a processor in the AI device, an input sequence of tokens, appending one or more mask tokens to the input sequence of tokens to generate a modified input token sequence, and inputting the modified input token sequence to a draft AI model, the draft AI model including a subset of layers of a target AI model. Further, the method can include generating, by the draft AI model, one or more draft tokens based on the modified input token sequence, verifying the one or more draft tokens, by the target AI model, to generate at least one accepted token, and generating an updated sequence of tokens by appending the at least one accepted token to the input sequence of tokens and outputting the updated sequence of tokens.