Neural Network Normalization Unit with Pre-computed Statistics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems face inefficiencies due to high data access speeds in batch memory, leading to increased training times and limitations in merging tasks across layers, particularly in batch normalization processes.
Innovation Solution
The proposed solution involves a neural network processing apparatus with a memory hierarchy that includes on-chip and off-chip memory blocks, a statistics determination unit, and a normalization unit, which allows for the prediction of statistics and normalization of tensors without relying on frequent data reads and writes, thereby streamlining the processing pipeline and reducing memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data access speed in batch memory is increased, then processing throughput is improved, but training time increases and task merging becomes limited
Solution Approach 1:
The batch normalization process is segmented into two distinct phases: a training phase where statistics are computed from mini-batches, and an inference phase where pre-computed statistics are applied to entire batches. This segmentation allows fast data access during inference without requiring repeated statistics computation, thereby reducing overall training time while maintaining high throughput.
Solution Approach 2:
Statistics (mean and variance) are pre-computed during the training phase from smaller mini-batches and stored for later use. This preliminary action eliminates the need to re-compute statistics during the inference phase, enabling fast batch processing without increasing training time, as the statistics are prepared in advance.
2Measurement precision
If statistics are determined from every batch, then normalization accuracy is improved, but data access frequency increases and processing efficiency decreases
Solution Approach 1:
Instead of computing statistics from every entire batch (excessive action), the system computes statistics from smaller mini-batches during training (partial action). This partial computation is sufficient to capture the essential distribution characteristics, providing adequate normalization accuracy while significantly reducing data access frequency and improving processing efficiency during inference.
Solution Approach 2:
Statistics are pre-computed from training mini-batches and cached for reuse during inference. This preliminary computation ensures that normalization accuracy is maintained based on representative training data, while avoiding repeated full-batch statistics computation that would reduce processing efficiency.
3Measurement precision
If batch normalization is performed for each layer separately, then processing accuracy is maintained, but device complexity and memory footprint increase
Solution Approach 1:
The patent merges the batch normalization operation with adjacent layer operations (such as convolution or fully connected layers) into a single unified processing step. This merging eliminates the need for separate memory allocations for intermediate normalization results, reducing memory footprint and device complexity while maintaining processing accuracy through the combined operation.
Data Source
AI summary
The present disclosure relates to neural network processing apparatus and method. An exemplary neural network processing apparatus includes: a memory for storing data; an operation unit to perform an operation on a first batch of tensors; a statistics determination unit communicatively coupled to the operation unit and configured to receive an output from the operation unit and determine statistics from the output; and a normalization unit communicatively coupled to the memory and the statistics determination unit and configured to receive a second batch of tensors from the memory and the statistics from the statistics determination unit and to normalize the second batch of tensors using the statistics.