Feature Map Reordering for Sparse Activation Memory Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly convolutional neural networks, face memory bandwidth bottlenecks and increased power consumption due to the large memory requirements for storing activation tensor data, which cannot be saved in on-chip memory, necessitating frequent transfers between on-chip and off-chip memory.
Innovation Solution
The solution involves sorting filters according to the sparsity of feature maps during the training stage, allowing for more efficient compression and storage of tensor data in memory by placing sequential zero values together, thereby reducing memory transfers during inference using formats like NHWC and compression algorithms such as run-length encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If activation tensor data is stored in off-chip memory, then memory capacity is sufficient, but memory bandwidth usage increases and power consumption increases
Solution Approach 1:
The patent applies preliminary action by sorting filters according to their sparsity levels during the training stage, before the inference stage begins. This pre-processing arrangement ensures that during inference, feature maps with higher sparsity (more zero values) are positioned to benefit from compression algorithms, thereby reducing memory bandwidth usage and power consumption without requiring changes to the underlying hardware or data storage capacity.
2Volume of stationary object
If activation tensor data is stored in off-chip memory, then memory capacity is sufficient, but memory transfer frequency increases
Solution Approach 1:
The patent changes the parameter of data organization by sorting filters based on sparsity metrics. This parameter change transforms the memory access pattern and data layout, enabling more effective compression during memory transfers. By altering how data is organized rather than how much is stored, the system reduces transfer frequency and improves productivity without sacrificing memory capacity.
3Loss of energy
If filters are sorted by sparsity during training, then compression efficiency improves, but training stage complexity increases
Solution Approach 1:
The patent performs the sorting operation during the training stage as a preliminary action, accepting increased training complexity in exchange for significant energy savings during the inference stage. This trade-off is justified because the energy reduction during deployment (inference) far exceeds the additional computational cost incurred during training, especially given that training is typically performed less frequently than inference operations.
Data Source
AI summary
A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a sparsity of the feature maps and store the plurality of different feature maps in the memory.