Distributed Batch Normalization for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed neural network training, using per-replica mean and variance for batch normalization reduces accuracy and effectiveness, as small per-device batch sizes fail to approximate global statistics, leading to suboptimal performance.
Innovation Solution
Implementing distributed batch normalization by computing and using distributed means and variances across sub-groups of devices, rather than relying solely on per-replica calculations, to normalize layer outputs and balance global statistics, thereby improving training accuracy with minimal additional communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If per-replica mean and variance are used for batch normalization in distributed training, then cross-device communication cost is minimized, but training accuracy deteriorates due to insufficient approximation of global statistics
Solution Approach 1:
The system segments the distributed training devices into multiple sub-groups, where each sub-group computes its own distributed statistics independently. This segmentation allows parallel computation across sub-groups, reducing the need for global synchronization and communication while still achieving better statistical approximation than per-replica methods.
Solution Approach 2:
The system merges the per-replica statistics from multiple devices within each sub-group to compute distributed mean and variance. By combining statistics across devices in a sub-group, the system achieves a better approximation of global statistics while limiting communication to only within-sub-group boundaries rather than global device communication.
2Productivity
If per-device batch size is kept small for parallel training, then training speed is improved, but batch normalization effectiveness deteriorates due to insufficient batch statistics
Solution Approach 1:
The system transitions from a single-dimension per-device batch normalization to a multi-dimensional approach by organizing devices into sub-groups. The distributed statistics are computed across the sub-group dimension, effectively increasing the batch size for statistics computation without sacrificing the parallel training speed benefits of small per-device batches.
3Measurement precision
If distributed statistics are computed across all devices, then approximation of global statistics is improved, but communication overhead increases significantly
Solution Approach 1:
The system segments all devices into multiple smaller sub-groups, where distributed statistics are computed independently within each sub-group. This segmentation reduces communication overhead by limiting data exchange to within-sub-group boundaries rather than requiring global device communication, while still providing improved statistical approximation over per-replica methods.
Data Source
AI summary
Methods, systems, and apparatus, including instructions encoded on storage media, for performing reduction of gradient vectors for distributed training of a neural network. One of the methods includes receiving, at each of the plurality of devices, a respective batch; performing, by each device, a forward pass comprising, for each batch normalization layer: generating, by each of the devices, a respective output of the corresponding other layer for each training example in the batch, determining, by each of the devices, a per-replica mean and a per-replica variance; determining, for each sub-group, a distributed mean and a distributed variance from the per-replica means and the per-replica variances for the devices in the sub-group; and applying, by each device, batch normalization to the respective outputs of the corresponding other layer generated by the device using the distributed mean and the distributed variance for the sub-group to which the device belongs.