Feature-Map Compression for Neural Network Training Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks with a large number of layers experiences performance bottlenecks due to high memory access demands for storing and retrieving intermediate data, particularly when a significant portion of this data consists of zero values after passing through activation functions, leading to inefficient memory throughput.
Innovation Solution
Implementing a reduction engine that performs atomic operations for compressing or removing zeros from intermediate data during forward propagation and restoring them during backward propagation, along with transpose operations, without storing the intermediate results in main memory, using a neural network processor and DMA controller.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If intermediate data is stored in main memory during forward propagation for re-use during backward propagation, then the training process can be completed, but the memory throughput is significantly reduced due to high access demands and large volume of zero values
Solution Approach 1:
The patent extracts and removes zero values from intermediate feature map data before storing in main memory. By identifying and eliminating redundant zero elements, the system reduces the volume of data requiring memory access while preserving the essential non-zero information needed for backward propagation, thus improving memory throughput without compromising training completion
Solution Approach 2:
Instead of storing all intermediate data and filtering during retrieval, the patent inverts the approach by pre-processing and removing zeros before storage. This inversion of the traditional store-then-filter paradigm reduces the memory burden from the outset, allowing faster access during backward propagation
2Reliability
If all intermediate data including zero values is transferred to and from main memory, then complete data is available for backward propagation, but the amount of data transferred is excessively large reducing training efficiency
Solution Approach 1:
The patent extracts zero values from intermediate feature maps before memory transfer, creating a compressed representation that contains only non-zero elements. This extraction process maintains the completeness of useful information while eliminating redundant data, thereby reducing transfer time and improving training efficiency without sacrificing the data needed for accurate backward propagation
Solution Approach 2:
The patent changes the parameter of data representation by transforming full-precision feature maps into a sparse format that tracks only non-zero elements and their positions. This parameter change from dense to sparse representation dramatically reduces the volume of data transferred between memory and processor while preserving all computationally relevant information
Data Source
AI summary
Systems and methods are provided to improve the memory throughput for storing and reading intermediate data computed by layers of a neural network during a training process. A compression operation can be performed by removing the zeros from the intermediate data and storing locations of the zeros before storing the intermediate data in the memory for a forward pass of the training process. The compressed data can be read from the memory for a backward pass of the training process and de-compressed by inserting zeros based on the stored locations. Additionally, a transpose operation can be performed before compression as a first atomic operation, or after de-compression as a second atomic operation.