Pre-normalized Neural Network Layers for Distributed Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Batch normalization in neural networks increases computational resource consumption during training, especially in distributed training frameworks, due to the need for computing normalization statistics across an entire batch of training inputs.
Innovation Solution
The implementation of either pre-normalized layers or regularization normalization layers in neural networks, which perform normalization without requiring operations over the batch dimension, thereby reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch normalization is used in neural networks, then performance is improved, but computational resource consumption increases significantly during training
Solution Approach 1:
The patent extracts the normalization function from the batch normalization context and implements it at the layer level instead. By computing normalization statistics independently for each layer without requiring batch-level operations, the invention removes the computational overhead associated with batch normalization while preserving the performance benefits of normalization.
Solution Approach 2:
The patent segments the normalization operation from the batch processing context and applies it independently to each layer. This segmentation allows normalization to be performed without accumulating statistics across the entire batch, thereby reducing computational resource consumption while maintaining performance improvements.
2Reliability
If batch normalization is used in distributed training frameworks, then performance is improved, but device complexity and synchronization overhead increase
Solution Approach 1:
The patent extracts the normalization computation from the batch-level operation and relocates it to the layer level. This extraction eliminates the need for synchronization across accelerators for batch statistic accumulation, thereby reducing device complexity and synchronization overhead in distributed training frameworks while maintaining performance benefits.
3Reliability
If batch normalization is used, then normalization performance is achieved, but training overhead increases due to non-local memory-access patterns
Solution Approach 1:
The patent extracts the normalization statistics computation from the batch dimension and performs it locally at each layer. This extraction eliminates non-local memory-access patterns associated with batch normalization, thereby reducing training overhead and improving training efficiency while maintaining normalization performance.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using a neural network system that includes one or more pre-normalized layers or one or more regularization normalization layers.