Batch Normalization Accelerator with On-Chip Statistical Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network accelerators face challenges in scalability and high off-chip memory access costs during batch normalization operations in deep neural network training, leading to increased power consumption and processing time.
Innovation Solution
The proposed solution involves a many-core structured accelerator with core modules performing convolution operations, local reduction modules generating local statistical values, a global reduction module generating global statistical values, and a normalization module applying batch normalization using these values, optimized through a batch normalization fission-n-fusion (BNFF) process to reduce off-chip memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch normalization operations are performed in existing neural network accelerators, then normalization can be applied to feature map data, but off-chip memory access costs increase significantly
Solution Approach 1:
The batch normalization operation is segmented into two distinct phases: a training phase where statistical values (mean and variance) are computed and stored in on-chip memory, and an inference phase where these pre-computed values are reused. This segmentation eliminates repeated off-chip memory accesses during normalization operations, reducing memory access costs while maintaining normalization accuracy.
Solution Approach 2:
Statistical values (mean and variance) are pre-computed during the training phase and stored in on-chip memory before the actual normalization operations are performed. This preliminary action allows subsequent normalization operations to use these pre-computed values without requiring additional off-chip memory accesses, thereby reducing energy consumption and improving efficiency.
2Reliability
If batch normalization operations are performed in existing neural network accelerators, then feature map data can be normalized, but processing time increases
Solution Approach 1:
The batch normalization operation is segmented into a training phase for statistical value computation and an inference phase for reuse. This segmentation allows the system to avoid repeated computation and off-chip memory accesses during inference, significantly reducing processing time while maintaining normalization accuracy.
Solution Approach 2:
Statistical values are pre-computed and cached in on-chip memory during the training phase. This preliminary computation eliminates the need for repeated calculations and off-chip memory accesses during subsequent normalization operations, thereby reducing processing time without compromising accuracy.
3Productivity
If neural network training is scaled up, then model performance improves, but scalability is limited by memory access costs
Solution Approach 1:
The accelerator architecture is segmented into distinct functional units: convolution processing units, statistical value computation units, and normalization processing units. Each unit has dedicated on-chip memory, allowing independent operation and reducing inter-unit memory access. This segmentation enables scalable deployment as model complexity increases without proportionally increasing memory access bottlenecks.
Solution Approach 2:
The patent introduces a temporal dimension to the batch normalization process by separating training-time statistical computation from inference-time normalization application. This dimensional separation allows the system to scale processing throughput by reusing pre-computed statistical values across multiple inference operations, effectively decoupling model complexity from memory access requirements.
Data Source
AI summary
A device and method with batch normalization are provided. An accelerator includes: core modules, each core module including a respective plurality of cores configured to perform a first convolution operation using feature map data and a weight; local reduction operation modules adjacent to the respective core modules, each including a respective plurality of local reduction operators configured to perform a first local operation that obtains first local statistical values of the corresponding core module; a global reduction operation module configured to perform a first global operation that generates first global statistical values of the core module based on the first local statistical values of the core modules; and a normalization operation module configured to perform a first normalization operation on the feature map data based on the first global statistical values.