Batch Normalization Layer Training Using Pre-computed Statistics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing batch normalization methods require significant operational processing capability and storage space, leading to processing delays in on-device neural network learning due to the need for calculating and storing statistic values by mini-batch units.
Innovation Solution
A batch normalization layer training method that sets the gradients for the standard deviation and mean of the loss function to zero, allowing the use of pre-computed normalized statistic values from initial or previous neural networks, reducing the need for real-time calculation and storage of current mini-batch statistics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch normalization is performed by calculating statistic values (mean and standard deviation) for each mini-batch unit, then the learning convergence speed is improved and internal covariate shift is reduced, but the operational processing capability requirement increases and storage space requirement increases
Solution Approach 1:
The patent applies preliminary action by pre-calculating the statistic values (mean and standard deviation) for the entire training dataset before the actual training process begins. These pre-computed statistics are then reused during training instead of recalculating them for each mini-batch, thereby reducing real-time computational load while maintaining the benefits of batch normalization for learning convergence.
2Productivity
If batch normalization is performed by calculating statistic values for each mini-batch unit, then the learning convergence speed is improved, but the storage space requirement increases
Solution Approach 1:
The patent pre-calculates and stores statistic values for the entire training dataset before training begins. This preliminary computation allows the model to reuse these statistics during training without needing to store or recalculate mini-batch specific statistics, thereby reducing storage requirements during the actual training process while maintaining fast convergence.
3Measurement precision
If batch normalization calculates statistic value by mini-batch unit, then the normalization accuracy for each batch is improved, but the learning delay increases because learning is not immediately performed
Solution Approach 1:
The patent resolves this contradiction by performing the statistic calculation in advance for the entire dataset. This preliminary action eliminates the need to wait for mini-batch processing during training, as the normalization parameters are already available. The approach maintains normalization accuracy by using comprehensive dataset statistics while eliminating learning delays caused by real-time mini-batch statistic computation.
Data Source
AI summary
Disclosed is a batch normalization layer training method, which may be used in a neural network learning apparatus having limited operational processing capability and storage space. A batch normalization layer training method according to an embodiment of the present disclosure may perform batch normalization transform by setting the gradients of the standard deviation and the mean of the loss function to zero, and applying a normalized statistic value obtained from an initial neural network or a previous neural network to the gradient of the loss function. The neural network learning apparatus of the present disclosure may be connected or converged with an Artificial Intelligence module, an Unmanned Aerial Vehicle (UAV), a robot, an Augmented Reality (AR) apparatus, a Virtual Reality (VR), a 5G network service-related apparatus, etc.