BoS Token Cache for Quantized LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autoregressive language models experience outliers when the begin of sentence (BoS) token is input, due to large activation values, which are exacerbated by model quantization.
Innovation Solution
The proposed method generates a BoS cache or a fixed sequence cache using the machine learning model before or after quantization, and then executes inference using the quantized model, inputting the next token and the cache to maintain accuracy and avoid outliers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If model quantization is performed to reduce computational and memory costs, then energy consumption and memory usage are reduced, but outliers occur due to large activation values from BoS tokens
Solution Approach 1:
The patent applies preliminary action by precomputing the activation values for the BoS token before quantization and storing them in a cache. This precomputation allows the quantized model to use these precomputed values during inference, avoiding the generation of large activation values that cause outliers while maintaining the energy efficiency benefits of quantization.
Solution Approach 2:
The patent introduces a cache as an intermediary component that stores precomputed activation values for the BoS token. This cache acts as a mediator between the quantized model and the BoS token processing, allowing the model to retrieve precomputed values instead of generating new large activation values during quantized inference, thereby reducing outliers while maintaining computational efficiency.
2Quantity of substance
If model quantization is performed to reduce memory storage, then memory usage is reduced, but accuracy deteriorates due to low-precision data types
Solution Approach 1:
The patent applies local quality by using different precision levels for different parts of the model processing. Specifically, the BoS token activation values are computed and stored in high precision (float32) in the cache, while the rest of the model operations use low precision (int8). This allows the system to maintain high accuracy for the critical BoS token processing while achieving memory efficiency through quantization for the remaining computations.
3Productivity
If the BoS token is processed through the quantized model, then inference can proceed, but large activation values cause outliers and reduce output quality
Solution Approach 1:
The patent applies preliminary action by precomputing the activation values for the BoS token before quantization and storing them in a cache. This precomputation allows the quantized model to use these precomputed values during inference, avoiding the generation of large activation values that cause outliers while maintaining the energy efficiency benefits of quantization.
Solution Approach 2:
The patent extracts the BoS token processing from the main quantized model inference path by precomputing its activation values separately and storing them in a cache. This extraction allows the rest of the model to operate efficiently with quantized operations while the BoS token uses precomputed high-precision values, preventing outliers without sacrificing inference speed.
Data Source
AI summary
An execution method of a machine learning model, comprising: generating output and a begin of sentence (BoS) cache of a BoS token using the machine learning model before or after performing model quantization on the machine learning model to generate a quantized model; and executing inference based on the quantized model, and during the inference, input the next token following the BoS token as a first input token and the BoS cache into the quantized model to generate output and cache of the next token, wherein the next token is based on the output of the Bos token or based on an input content.