Transformer Quantization with Adaptive Rounding for Accuracy Retention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional model quantization schemes optimize only quantization factors, leading to significant performance loss and reduced accuracy in task processing using quantized models, particularly in Transformer models, which require substantial computing resources.
Innovation Solution
Optimize both quantization factors and rounding directions in Transformer models by minimizing differences between quantized and floating-point outputs, using adaptive rounding methods like round-up or round-down to enhance model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If traditional model quantization scheme is used to reduce resource consumption, then storage and computational resources are reduced, but model performance loss increases and accuracy decreases
Solution Approach 1:
The patent applies parameter changes by optimizing quantization factors and rounding directions to minimize the difference between quantized and floating-point outputs. This involves adjusting the quantization parameters (scale factors and zero-points) and selecting optimal rounding directions (round-up or round-down) for different layers and operations, thereby maintaining model performance while using lower precision data formats
Solution Approach 2:
The patent introduces dynamic optimization by adaptively determining rounding directions based on the specific characteristics of each layer and operation in the Transformer model. Instead of using a fixed rounding scheme, the method dynamically selects rounding strategies during the quantization process to minimize performance loss in different contexts
2Measurement precision
If 32-bit floating-point numbers are used to maintain model accuracy, then task processing accuracy is maintained, but resource consumption and computational cost increase
Solution Approach 1:
The patent changes the precision parameter from 32-bit floating-point to lower precision formats (such as 8-bit or 4-bit integers) while compensating for the loss of precision through optimized quantization factors and rounding directions. This allows maintaining acceptable accuracy levels with significantly reduced storage requirements
3Productivity
If lower-precision data formats are used to enhance model speed, then inference speed is improved, but performance loss increases
Solution Approach 1:
The patent optimizes the quantization parameters including scale factors and zero-points to minimize the performance gap between low-precision and floating-point models. By carefully tuning these parameters and selecting appropriate rounding directions, the method maintains model performance while achieving faster inference speeds
Solution Approach 2:
The patent performs preliminary optimization of quantization factors and rounding directions during the model preparation phase. This preliminary action ensures that when the quantized model is deployed for fast inference, the performance loss is already minimized, allowing the model to achieve both speed and accuracy
Data Source
AI summary
A task processing method includes: updating, based on a first difference between a first quantized output and a first floating-point output of an optimization unit in a Transformer model, a weight quantization factor and an activation quantization factor of the optimization unit; updating, based on a second difference between a second quantized output and a second floating-point output of the optimization unit, a weight quantization increment of the optimization unit; determining a weight quantization rounding direction of the optimization unit based on the target weight quantization increment; and quantizing the weight parameter of the optimization unit based on the target weight quantization factor and the weight quantization rounding direction; performing forward inference computation on input data of the optimization unit based on a target quantization weight parameter for the optimization unit, and quantizing an input/output of the optimization unit based on a target activation quantization factor of the optimization unit.