Quantization Parameter Storage for Faster Large Model Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high cost and resource-intensive nature of large model inference, particularly due to the large number of parameters and context, leads to significant memory and computing requirements, limiting scalability and efficiency.
Innovation Solution
A method for calculating and storing quantization parameters in advance, using statistical values from benchmark data to determine target values, which are then used for quantizing and dequantizing key value matrices during inference, reducing the need for repeated calculations and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If quantization parameters are calculated during the inference process, then the inference accuracy is maintained, but the memory occupancy and computing resources increase significantly
Solution Approach 1:
The patent applies preliminary action by calculating and storing quantization parameters (scale and zero-point values) before the inference process. The system performs statistical analysis on benchmark data to determine optimal quantization parameters, then stores these pre-calculated values in a parameter table. During inference, the system directly retrieves these pre-stored parameters instead of recalculating them, thereby reducing memory occupancy and computing resources while maintaining inference accuracy.
2Reliability
If quantization parameters are calculated during the inference process, then the inference accuracy is maintained, but the inference speed decreases due to repeated calculations
Solution Approach 1:
The patent applies preliminary action by calculating and storing quantization parameters (scale and zero-point values) before the inference process. The system performs statistical analysis on benchmark data to determine optimal quantization parameters, then stores these pre-calculated values in a parameter table. During inference, the system directly retrieves these pre-stored parameters instead of recalculating them, thereby reducing memory occupancy and computing resources while maintaining inference accuracy.
3Quantity of substance
If low-bit quantization is applied to reduce memory usage, then the memory occupancy decreases, but the model inference cost increases due to additional quantization and dequantization operations
Solution Approach 1:
The patent applies preliminary action by calculating and storing quantization parameters (scale and zero-point values) before the inference process. The system performs statistical analysis on benchmark data to determine optimal quantization parameters, then stores these pre-calculated values in a parameter table. During inference, the system directly retrieves these pre-stored parameters instead of recalculating them, thereby reducing memory occupancy and computing resources while maintaining inference accuracy.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting quantization parameters (scale and zero-point values) based on statistical analysis of benchmark data. The system calculates mean and standard deviation values from the data distribution, then uses these statistical parameters to determine optimal quantization settings. This allows the system to adapt quantization parameters to the specific characteristics of the input data, reducing the need for complex runtime quantization operations while maintaining accuracy.
Data Source
AI summary
Provided is a quantization parameter storage method, a model inference method, an electronic device and a storage medium, relating to the fields of large model technology, artificial intelligence technology and model quantization technology. The quantization parameter storage method includes: obtaining, by a calculation unit of a processor, a statistical value of a first quantization parameter of a model statistically based on benchmark data; searching for, by the calculation unit, a target value of the first quantization parameter and a target value of a second quantization parameter of the model in a search space based on the statistical value of the first quantization parameter; and storing, by the calculation unit, the target value of the first quantization parameter and the target value of the second quantization parameter into a memory.