Low-Bit Model Quantization With Iterative Distillation Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model quantization technologies result in significant precision loss when converting floating-point computing models to low-bit fixed-point computing models.
Innovation Solution
A method involving distillation training using an initial model as a teacher model for a quantization model, iteratively reducing the quantization bit width until a target threshold is reached, with each iteration using the previous model's bit width for training to improve precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model quantization is performed to reduce computing intensity and parameter size, then the model can be deployed more efficiently, but precision loss occurs in the quantization model
Solution Approach 1:
The patent applies preliminary action by performing distillation training before final low-bit quantization. The method first quantizes the model to a intermediate bit width, trains the quantization model using the original model as teacher, and then performs final quantization. This preliminary training phase prepares the quantization model to better preserve precision during subsequent low-bit quantization, directly addressing the precision loss problem while maintaining productivity benefits.
2Ease of operation
If the quantization bit width is reduced to improve model efficiency, then deployment becomes more practical, but the precision of the quantization model deteriorates
Solution Approach 1:
The patent performs distillation training at an intermediate quantization bit width before final low-bit quantization. This preliminary action allows the quantization model to learn from the original model's knowledge while still maintaining reasonable precision requirements for deployment. The method thus enables practical deployment with reduced bit width while mitigating precision deterioration through the preliminary training phase.
Solution Approach 2:
The patent introduces an intermediate quantization bit width as a mediator between the original high-precision model and the final low-bit quantization model. This intermediate model serves as a bridge, allowing knowledge distillation to occur at a precision level that is more manageable than the original model but still sufficiently precise to guide the final low-bit quantization. This intermediary approach balances deployment efficiency with precision preservation.
3Quantity of substance
If conventional quantization methods are used to reduce parameter size, then memory consumption decreases, but huge precision loss is inevitable
Solution Approach 1:
The patent applies preliminary distillation training at an intermediate quantization level before final low-bit quantization. This preliminary action allows the quantization model to adapt to quantization effects while retaining sufficient precision to learn from the original model. As a result, the final low-bit model achieves better precision retention despite significant parameter size reduction, directly addressing the huge precision loss problem of conventional methods.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
The present disclosure relates to a model quantization method and apparatus, a medium, and an electronic device, and relates to the field of computer technologies. The method includes: quantizing an initial model according to an initial quantization bit width to obtain a quantization model; using the initial model as a teacher model of the quantization model, and performing distillation training on the quantization model to obtain a trained quantization model; reducing the initial quantization bit width to obtain a target quantization bit width; and using the target quantization bit width as the initial quantization bit width and using the trained quantization model as the initial model, and continuing to quantize the initial model until the target quantization bit width reaches a target threshold, and determining the trained quantization model corresponding to the target quantization bit width that reaches the target threshold as a complete-trained quantization model. A model with a previous quantization bit width can be used as a teacher model of a current quantization model, and the distillation training is performed on the current quantization model, so that it is ensured that the precision of the quantization model can be further improved.