Residual Connection Quantization for Accuracy-Latency Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional model quantization methods that reduce precision for both weights and activations lead to accuracy degradation due to the inability to represent outlier values, while quantizing only weights maintains unsatisfactory latency improvements.
Innovation Solution
A mixed-precision model quantization method that assigns higher precision to activations in residual connections and lower precision to other activations within a neural network model, balancing accuracy and latency by preserving outlier information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If full quantization with low precision is applied to both weights and activations, then latency is reduced and computational efficiency is improved, but model accuracy deteriorates due to inability to represent outlier values
Solution Approach 1:
The patent applies different precision levels to different parts of the model based on their specific needs. Activations in residual connections are assigned higher precision (first precision) to accurately represent outlier values, while activations in non-residual paths use lower precision (second precision). This local differentiation resolves the contradiction by maintaining accuracy where needed while achieving latency reduction elsewhere.
Solution Approach 2:
The patent segments the model's activation flows into two distinct groups: those passing through residual connections and those that don't. By separating activations into these segments and applying different precision settings to each segment, the system achieves both accuracy preservation (for residual paths) and latency reduction (for non-residual paths).
2Manufacturing precision
If quantization is applied only to weights while maintaining full precision for activations, then model accuracy is maintained, but latency reduction is insufficient
Solution Approach 1:
Instead of uniformly maintaining full precision for all activations, the patent selectively applies higher precision only to activations in residual connections where it is most needed for accuracy. Other activations use lower precision, achieving latency reduction while maintaining overall model accuracy through targeted precision application.
3Productivity
If low precision is used for activations, then computational demands are reduced, but the ability to represent the full range of activation values including outliers is lost
Solution Approach 1:
The patent assigns different precision levels to different activation groups based on their representation needs. Activations in residual connections receive higher precision (first precision) to accurately represent the full range of values including outliers, while activations in other paths use lower precision (second precision) to maximize computational efficiency. This resolves the contradiction by matching precision levels to actual representation requirements.
Data Source
AI summary
A mixed-precision model quantization method includes loading a trained model, and quantizing the trained model with a mixed-precision setting to generate a quantized model for inference. The trained model includes a plurality of residual connections. In each residual connection, a first activation bypasses at least one operator and is added to a second activation to generate a fourth activation. The second activation is the output of the first activation after being processed by the at least one operator, The mixed-precision setting includes (a) the first activation, the second activation, and the fourth activation in at least one residual connection of the plurality of residual connections being assigned a first precision, and (b) third activations in all operators bypassed by the at least one residual connection being assigned a second precision. The third activations are generated by the bypassed operators and processed within the bypassed operators.