Quantization Model Optimization via Clipping Range Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models face accuracy degradation due to quantization loss when transitioning from 32-bit floating-point to fewer bits, and existing finetuning methods require significant computational resources and time, especially when adapting to varying environmental conditions.
Innovation Solution
A method to optimize quantization models by recalibrating quantization parameters without modifying the deep learning compiler's internal code, allowing for environment-adaptive models by adjusting clipping ranges and recomputing scale factors and zero points, thereby improving accuracy without extensive retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If quantization is applied to reduce computational resources and model size, then the model can run on embedded devices with limited resources, but quantization loss occurs causing degradation in model accuracy
Solution Approach 1:
The patent changes the quantization parameters (clip min and clip max values) to optimize the balance between model size and accuracy. By adjusting these parameters, the system can control the quantization range and distribution, thereby reducing quantization loss while maintaining the benefits of reduced computational resources.
Solution Approach 2:
The patent performs preliminary calibration by collecting calibration data and computing optimal quantization parameters before deploying the quantized model. This preliminary action allows the system to pre-determine the best quantization settings for specific environments, avoiding accuracy degradation during actual inference.
2Reliability
If finetuning is performed to adapt the model to different environments and maintain accuracy, then model performance can be maintained, but significant computational resources and time are required for retraining
Solution Approach 1:
Instead of performing full finetuning which requires extensive retraining, the patent changes only the quantization parameters (clip min and clip max) based on calibration data from the target environment. This parameter adjustment approach achieves environment adaptation with minimal computational resources and time.
Solution Approach 2:
The patent collects calibration data and computes optimal quantization parameters in advance before deploying the model to the target environment. This preliminary calibration action enables the model to adapt to different environments without requiring time-consuming retraining during deployment.
3Ease of operation
If the quantization process is preconfigured in the deep learning compiler, then users can easily perform quantization, but it is very difficult for users to directly implement custom quantization modifications
Solution Approach 1:
The patent separates the quantization process into two independent parts: (1) the preconfigured quantization process in the deep learning compiler that users can easily access, and (2) the post-processing step that applies custom quantization parameter adjustments. This segmentation allows users to benefit from both the ease of using preconfigured quantization and the flexibility of custom modifications.
Solution Approach 2:
The patent introduces an intermediary calibration step that acts as a bridge between the preconfigured quantization process and custom modifications. The calibration process collects data from the target environment and computes optimal parameters, serving as an intermediary that enables custom adaptations without requiring users to modify the complex compiler internals.
4Reliability
If a model is trained using training data with a specific distribution, then the model achieves good training performance, but performance degrades when the inference environment has a different data distribution
Solution Approach 1:
The patent performs preliminary calibration by collecting calibration data from the target inference environment and computing optimal quantization parameters before deployment. This preliminary action allows the model to adapt to the specific data distribution of the target environment, bridging the gap between training and inference distributions.
Solution Approach 2:
The patent adjusts the quantization parameters (clip min and clip max) based on the statistical characteristics of the calibration data from the target environment. By changing these parameters to match the target environment's data distribution, the model maintains high performance despite distribution differences between training and inference.
Data Source
AI summary
Disclosed is a method and system for optimizing a quantization model. A quantization model optimization method may include receiving an input of the quantization model; extracting at least one of a weight and an activation, and a quantization parameter of the at least one of the weight and the activation by analyzing the input quantization model; selecting at least one of the weight and the activation of the input quantization model as a target element to be modified; adjusting a clipping range related to the quantization parameter of the target element; recomputing the quantization parameter of the target element based on the adjusted clipping range; and generating an adjusted quantization model by applying the recomputed quantization parameter to the input quantization model.