Quantizing Neural Networks with Batch Normalization Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks for quantized inference with batch normalization is challenging due to jitter introduced by variations in batch statistics, leading to poor inference performance and resource inefficiency.
Innovation Solution
The system applies a correction factor to floating-point weights for batch normalized layers, switching from batch statistics to long-term moving averages after sufficient training, which stabilizes layer outputs and reduces jitter, enabling efficient quantization and improved performance during inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If batch normalization uses batch statistics during training, then normalization adapts to current batch distribution, but jitter is introduced in quantized weights during training
Solution Approach 1:
The patent pre-computes normalization statistics from the entire training dataset before the actual training process begins. These pre-computed statistics are then used throughout training instead of computing batch statistics during each training iteration. This preliminary action eliminates the jitter problem while maintaining the benefits of normalization.
Solution Approach 2:
The patent introduces an intermediate step of pre-computing statistics from the full training dataset. This intermediate statistical representation serves as a mediator between the training data and the normalization process, providing stable reference statistics that eliminate jitter while still capturing the data distribution characteristics.
2Productivity
If conventional batch normalization is used during training, then normalization is computed efficiently per batch, but outputs during training differ from inference outputs
Solution Approach 1:
The patent performs the statistics computation in advance using the entire training dataset before training begins. This preliminary computation ensures that the same stable statistics are used during both training and inference, eliminating the discrepancy between training and inference outputs while maintaining training efficiency.
3Use of energy by moving object
If quantized inference is used, then computational resources are reduced, but training for quantized inference is challenging with batch normalization
Solution Approach 1:
The patent pre-computes normalization statistics from the entire training dataset before training begins. This preliminary action creates stable reference statistics that eliminate jitter during quantized training, making training easier while preserving the resource efficiency benefits of quantized inference.
Solution Approach 2:
The patent converts the potential harm of quantization jitter into a benefit by using pre-computed statistics. The same quantization process that would normally cause jitter is instead stabilized by the pre-computed statistics, turning the challenge of quantized training into an opportunity for improved performance.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network that has one or more batch normalized neural network layers for use by a quantized inference system. One of the methods includes receiving a first batch of training data; determining batch normalization statistics for the first batch of training data; determining a correction factor from the batch normalization statistics for the first batch of training data and the long-term moving averages of the batch normalization statistics; generating batch normalized weights from the floating point weights for the batch normalized first neural network layer, comprising applying the correction factor to the floating point weights of the batch normalized first neural network layer; quantizing the batch normalized weights; determining a gradient of an objective function; and updating the floating point weights using the gradient.