Stochastic Whitening Batch Normalization for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face challenges in training convergence and generalization due to computational inefficiencies, particularly with methods like Iterative Normalization, which require expensive matrix decompositions and inversions at each training step, leading to high computational overhead and limited scalability.
Innovation Solution
The stochastic whitening batch normalization (SWBN) technique gradually learns whitening matrices separately in each layer during training, decoupling the whitening loss from the task loss and using internal matrices to incrementally whiten and rescale activations, reducing the need for expensive matrix operations and improving convergence and generalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Iterative Normalization is used to whiten activations at each training step, then generalization performance is improved, but computational overhead increases significantly due to expensive matrix decompositions and inversions
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the whitening matrix during a separate training phase. Instead of performing expensive matrix decompositions and inversions at each training step, the whitening matrix is computed once in advance and then reused during subsequent training iterations, significantly reducing computational overhead while maintaining generalization performance
Solution Approach 2:
The patent uses copying by creating a pre-computed whitening matrix that is copied and applied repeatedly during training. Rather than recalculating the whitening transformation at each step, the system creates the transformation matrix once and copies its application across multiple training iterations, eliminating redundant computational operations
2Measurement precision
If Newton's method is used to compute exact whitening matrix at each training step, then data whitening accuracy is improved, but training speed decreases due to multiple iterations of matrix multiplications
Solution Approach 1:
The patent applies preliminary action by performing the computationally intensive Newton's method iterations and matrix decompositions in advance during a preprocessing phase. The exact whitening matrix is computed once with high accuracy, then this pre-computed matrix is reused during actual training, achieving both high whitening accuracy and fast training speed
Solution Approach 2:
The patent segments the training process into two distinct phases: a preliminary phase for computing the exact whitening matrix using Newton's method, and a main training phase for applying the pre-computed matrix. This segmentation allows the computationally expensive operations to be performed once rather than repeatedly, improving overall training speed while maintaining whitening accuracy
3Speed
If data whitening is applied to all layers of deep neural network, then convergence rate is improved, but computational complexity increases due to repeated matrix operations at each layer
Solution Approach 1:
The patent applies universality by creating a single whitening matrix that serves all layers of the neural network. Instead of computing separate whitening matrices for each layer, the system computes one universal whitening matrix that is applied across all layers, reducing computational complexity while still improving convergence rate through whitening
Solution Approach 2:
The patent merges the whitening operation into a single unified transformation that is applied once before data enters the network. Instead of performing separate whitening operations at each layer, the system combines all whitening transformations into one operation, reducing the number of matrix operations from O(L) to O(1) where L is the number of layers
Data Source
AI summary
A neural network system, comprising: instructions for implementing at least a SWBN layer in a neural network, and wherein the instructions perform operations comprising: during training of the neural network system on a plurality of batches of training data and for each of the plurality of batches: obtaining a respective first layer output for each of the plurality of training data; determining a plurality of normalization statistics for the batch from the first layer outputs; generating a respective normalized output for each training data in the batch; updating the whitening matrix by a covariance matrix; performing stochastic whitening on the normalized components of each first layer output; transforming the whitened data for each training data; generating a respective SWBN layer output for each of the training data from the transformed whitened data for each training data in the batch; and providing the SWBN layer output.