Distributed Batch Normalization for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data parallel training of neural networks, the synchronization issue caused by calculating normalization parameters based on statistical measures from multiple processors significantly slows down the training process, as each processor stalls waiting for responses from other processors, leading to inefficiencies in processing and increased training time.
Innovation Solution
The method involves generating statistical measures from partial populations of training data across multiple processors, reducing these measures to generate normalization parameters, and sharing them to normalize activations, allowing processors to continue processing additional samples while waiting for normalization parameters, thus reducing latency and increasing training speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normalization parameters are calculated based on statistical measures from multiple processors, then the normalization accuracy is improved, but the training speed deteriorates due to synchronization stalls
Solution Approach 1:
The patent applies preliminary action by having each processor calculate statistical measures (mean and variance) for its local batch portion before the synchronization point. This allows preprocessing of normalization data in advance, so that when normalization parameters are needed, the calculation can proceed more quickly using pre-computed statistics rather than computing everything at the synchronization moment.
Solution Approach 2:
The patent segments the batch normalization calculation by dividing the overall batch into portions handled by different processors. Each processor independently calculates statistical measures for its segment, then these segment statistics are aggregated to form the overall normalization parameters. This segmentation enables parallel computation while maintaining accuracy through proper aggregation of statistical measures.
2Reliability
If processors wait for normalization parameters from other processors, then the normalization parameters are accurate, but the processing time increases due to idle waiting
Solution Approach 1:
The patent implements continuity of useful action by allowing processors to perform productive work during what would otherwise be idle waiting time. Specifically, processors can begin processing the next batch of data or perform other computational tasks while waiting for normalization parameters from other processors, ensuring that computational resources remain continuously utilized rather than idle.
Solution Approach 2:
The patent applies preliminary action by having processors pre-calculate statistical measures for their local data portions before the synchronization point. This preliminary computation reduces the amount of work needed during the waiting period, and allows processors to use this pre-computed information productively while waiting for other processors' contributions to the overall normalization parameters.
3Measurement precision
If statistical measures are calculated for the entire batch, then the normalization parameters are accurate, but the calculation complexity increases
Solution Approach 1:
The patent segments the complex task of calculating statistical measures for the entire batch into smaller, independent segments that can be handled by individual processors. Each processor calculates mean and variance for its local portion of the batch, avoiding the complexity of having a single processor handle all data. The segment statistics are then aggregated using straightforward mathematical operations to obtain the overall normalization parameters.
Solution Approach 2:
The patent applies self-service by enabling each processor to independently calculate its own local statistical measures without requiring centralized coordination or complex inter-processor communication during the calculation phase. Each processor autonomously computes mean and variance for its local batch portion, then simply contributes these results to the overall normalization parameter calculation, reducing overall system complexity.
Data Source
AI summary
A technique for performing data parallel training of a neural network model is disclosed that incorporates batch normalization techniques using partial populations to generate normalization parameters. The technique involves processing, by each processor of a plurality of processors in parallel, a first portion of a sub-batch of training samples allocated to the processor to generate activations for the first portion of the sub-batch. Each processor analyzes the activations and transmits statistical measures for the first portion to an additional processor that reduces the statistical measures from multiple processors to generate normalization parameters for a partial population of the training samples that includes the first portion from each of the plurality of processors. The normalization parameters are then transmitted back to each of the processors to normalize the activations for both the first portion and a second portion of the sub-batch of training samples allocated to each processor.