Batch Norm Parameter Training for Quantization Error Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Simplified neural networks with depth-wise separable convolutions face performance degradation due to significant quantization errors caused by varying dynamic ranges of activation data across channels, especially during low-bit quantization recognition training.
Innovation Solution
A method is introduced to train batch norm parameters by calculating quantization errors using signal-to-quantization-noise ratio (SQNR) and updating them via stochastic gradient descent to reduce regularization loss, while fixing other parameters, thereby adjusting the activation distribution to match a single quantization scale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If depth-wise separable convolutions are used to simplify the neural network, then device complexity and computational cost are reduced, but quantization errors increase due to varying dynamic ranges across channels
Solution Approach 1:
The patent applies local quality by training separate batch normalization parameters (gamma and beta) for each channel independently. This allows each channel to have its own scaling and shifting characteristics, adapting to the specific dynamic range of activation data in that channel. The channel-wise batch norm parameters enable localized adjustment of quantization characteristics, resolving the contradiction between network simplification and quantization accuracy by treating each channel with its own optimized parameters rather than a uniform approach.
2Device complexity
If a single quantization scale is used across all channels, then device complexity is reduced, but performance degrades due to significant quantization errors from varying dynamic ranges
Solution Approach 1:
The patent employs parameter changes by introducing trainable batch normalization parameters (gamma and beta) that are optimized during training. These parameters dynamically adjust the scaling and shifting of activation data for each channel, effectively changing the quantization characteristics to match the actual data distribution. This resolves the contradiction by allowing the system to adapt parameters based on learned patterns rather than using fixed, uniform quantization scales.
Solution Approach 2:
The patent applies preliminary action by pre-training the batch normalization parameters before the actual quantization process. During this preliminary training phase, the gamma and beta parameters are optimized to minimize quantization errors. Once trained, these parameters are fixed and used during inference, providing pre-configured optimization that prevents quantization errors without adding complexity to the quantization process itself.
3Manufacturing precision
If batch norm parameters are trained to reduce quantization errors, then quantization accuracy improves, but training time and computational resources increase
Solution Approach 1:
The patent applies partial action by training only the batch normalization parameters (gamma and beta) while keeping other network parameters fixed during the quantization optimization phase. This selective training approach focuses computational resources only on the parameters that directly affect quantization accuracy, rather than retraining the entire network. This resolves the contradiction by achieving quantization accuracy improvement with minimal additional training time and computational overhead.
Data Source
AI summary
Disclosed is a processor implemented method that includes calculating a quantization error for each channel of a neural network using activation data output from a first layer of the neural network and a quantization scale of a second layer connected to the first layer, calculating a final loss using a regularization loss term determined based on the quantization error for each channel, and updating a batch norm parameter of the first layer in a direction to decrease the final loss.