Machine Learning Normalization Pipelining for Lower Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning models, particularly those using neural networks and transformer-based models, face significant computational complexity and expense due to repeated data accesses and processing required for normalization operations like layer normalization, which are performed individually for each segment of the feature tensor, leading to substantial computational cost and latency.
Innovation Solution
An improved architecture and method for layer normalization that utilizes pre-processing and post-processing logic within a single processing unit iteration to generate normalization scale factors, reducing data bandwidth and processing time by pipelining tensor segments, thereby minimizing memory accesses and cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normalization operations are performed individually for each segment of the feature tensor using conventional methods, then the normalization accuracy is maintained, but the computational complexity and processing time increase substantially
Solution Approach 1:
The patent combines multiple normalization operations into a single processing unit iteration. Instead of processing each segment separately through multiple memory accesses, the system generates mean values, intermediate segments, and normalization scaling factors for multiple segments simultaneously within one iteration, merging what were previously sequential operations into a parallelized unified process.
Solution Approach 2:
The patent performs preliminary actions by generating mean values and intermediate segments in advance within the same processing iteration. The system calculates these values beforehand so that when normalization scaling factors are needed, they are already available in registers, eliminating the need for additional memory accesses and enabling faster processing while maintaining accuracy.
2Reliability
If conventional normalization operations are used with repeated data accesses from off-die memory, then the normalization operation can be completed, but the bandwidth usage and processing latency increase significantly
Solution Approach 1:
The patent segments the feature tensor into multiple segments that can be processed in parallel within a single iteration. By dividing the data into manageable segments and processing them simultaneously with appropriate pipelining, the system reduces the need for repeated memory accesses while ensuring each segment receives the proper normalization treatment, thus completing the operation reliably with reduced latency.
Solution Approach 2:
The patent ensures continuity of useful action by pipelining the processing of multiple segments through different stages (mean generation, intermediate segment creation, scaling factor calculation) within the same iteration. This continuous processing eliminates idle time and repeated memory accesses, maintaining operational reliability while significantly reducing processing latency.
3Reliability
If traditional layer normalization is implemented with four separate data accesses to off-die memory, then the normalization can be performed, but the computational cost and bandwidth consumption increase by substantial margins
Solution Approach 1:
The patent merges four separate data access operations into a single processing iteration. By combining the generation of mean values, intermediate segments, and normalization scaling factors for multiple segments into one unified operation that utilizes on-die memory, the system maintains normalization correctness while dramatically reducing bandwidth consumption and computational cost.
Solution Approach 2:
The patent introduces on-die memory as an intermediary between the processing unit and off-die memory. This intermediary allows the system to store and retrieve intermediate results (mean values, intermediate segments) quickly during the same iteration, eliminating the need for repeated off-die memory accesses and reducing both computational cost and energy consumption while preserving normalization correctness.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, an input tensor comprising a plurality of segments is accessed, the input tensor generated while processing data using a machine learning model. A normalization operation of the machine learning model is applied to the input tensor by generating a mean value for a segment of the input tensor, generating an intermediate segment based on differences between the mean value and each element of the segment, generating a normalization scaling factor for the segment based on the intermediate segment, generating a scaled segment based on scaling each element of the intermediate segment using the normalization scaling factor, and generating a normalized output tensor based on at least the scaled segment.