Iterative Model Quantization With Distillation for Low-Bit Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model quantization methods inevitably result in significant precision loss when converting floating-point computing models to low-bit fixed-point computing models.
Innovation Solution
A method involving quantizing an initial model based on an initial quantization bit width, using it as a teacher model for distillation training, reducing the bit width to a target threshold through iterative distillation, and using the trained model as the initial model until the target threshold is reached, ensuring precision is maintained.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model quantization is performed to reduce computing intensity and memory consumption, then computing efficiency is improved, but model precision deteriorates
Solution Approach 1:
The quantization process is segmented into multiple iterative stages, each reducing the bit width gradually (e.g., from 32-bit to 16-bit, then to 8-bit, and finally to 4-bit). This stepwise segmentation allows the model to adapt to each quantization level, minimizing precision loss at each stage while achieving the ultimate goal of low-bit quantization for improved computing efficiency.
Solution Approach 2:
Before performing full quantization to the target bit width, the method performs preliminary quantization to an intermediate bit width and conducts distillation training at each intermediate stage. This preliminary action prepares the model for subsequent lower-bit quantization, ensuring that precision is maintained throughout the quantization process while enabling final low-bit deployment for higher computing efficiency.
2Quantity of substance
If quantization bit width is reduced to decrease parameter size, then storage requirements are reduced, but model accuracy deteriorates
Solution Approach 1:
The parameter size reduction is achieved through segmented quantization stages, where the bit width is reduced incrementally (32→16→8→4 bits) with distillation training at each stage. This segmentation allows the model to maintain accuracy while progressively reducing parameter size, ultimately achieving compact model storage without significant accuracy loss.
Solution Approach 2:
Distillation training is employed as a feedback mechanism at each quantization stage, where the higher-bit model serves as a teacher to guide the lower-bit model's training. This feedback loop ensures that the lower-bit model learns to compensate for quantization errors, maintaining model accuracy while achieving reduced parameter size for efficient storage and deployment.
3Manufacturing precision
If iterative distillation training is performed to maintain precision, then training time increases, but quantization quality improves
Solution Approach 1:
The training process is segmented into multiple iterative distillation stages, each corresponding to a specific bit width reduction. By dividing the overall quantization task into manageable segments with targeted distillation training at each stage, the method achieves high quantization quality while making the extended training time more manageable and efficient compared to single-stage quantization.
Data Source
AI summary
The present disclosure relates to a model quantization method, a medium, and an electronic device. The method includes: quantizing an initial model according to an initial quantization bit width to obtain a quantization model; using the initial model as a teacher model of the quantization model, and performing distillation training on the quantization model to obtain a trained quantization model; reducing the initial quantization bit width to obtain a target quantization bit width; and using the target quantization bit width as the initial quantization bit width and using the trained quantization model as the initial model, and continuing to quantize the initial model until the target quantization bit width reaches a target threshold, and determining the trained quantization model corresponding to the target quantization bit width that reaches the target threshold as a complete-trained quantization model.