Batch Renormalization Layer for Neural Network Training Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network systems face issues during training due to internal covariate shift, where the distribution of each layer's inputs changes as the parameters of the previous layer change, leading to reduced training effectiveness and mismatch between batch normalization layer outputs generated during training and after training.
Innovation Solution
The implementation of batch renormalization layers, which compute current batch normalization statistics and use moving normalization statistics to adjust affine transform parameters, allowing for normalization and renormalization of layer outputs to match the distribution of the entire dataset, thereby stabilizing the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch normalization layers are used during training, then training speed is improved, but the outputs during training do not match outputs after training due to internal covariate shift
Solution Approach 1:
The patent applies parameter changes by introducing affine transform parameters (scale and bias) that are learned during training to adjust the normalized outputs. These parameters are dynamically updated based on the moving normalization statistics, allowing the layer to adapt its output distribution to match both training and inference phases while maintaining fast training convergence.
Solution Approach 2:
The patent implements preliminary action by computing moving normalization statistics (moving mean and moving variance) in advance during the training phase. These pre-computed statistics are then used during inference to generate consistent outputs without requiring batch-level computation, ensuring that the layer behaves consistently across both training and inference while enabling fast training.
2Productivity
If conventional batch normalization is applied, then training effectiveness improves, but sensitivity to initialization increases
Solution Approach 1:
The patent applies dynamics by making the normalization statistics adaptive through the introduction of moving statistics that evolve during training. The affine transform parameters are dynamically adjusted based on the relationship between current batch statistics and moving statistics, allowing the layer to adapt to different initialization conditions and reduce sensitivity to initial parameter values while maintaining training effectiveness.
3Device complexity
If batch normalization statistics are computed from current batch only, then computational simplicity is maintained, but mismatch with overall data distribution occurs
Solution Approach 1:
The patent merges two sources of normalization statistics: the current batch statistics and the moving statistics computed from previous batches. By combining these through the affine transform mechanism, the layer achieves both computational simplicity (using efficient batch-level computation) and distribution matching accuracy (by incorporating information from the overall data distribution through moving statistics).
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing a neural network. In one aspect, the neural network includes a batch renormalization layer between a first neural network layer and a second neural network layer. The first neural network layer generates first layer outputs having multiple components. The batch renormalization layer is configured to, during training of the neural network on a current batch of training examples, obtain respective current moving normalization statistics for each of the multiple components and determine respective affine transform parameters for each of the multiple components from the current moving normalization statistics. The batch renormalization layer receives a respective first layer output for each training example in the current batch and applies the affine transform to each component of a normalized layer output to generate a renormalized layer output for the training example.