Feature Map Reordering for Neural Network Memory Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models, particularly convolutional neural networks, face significant memory bandwidth bottlenecks and increased power consumption due to large memory requirements for storing activation tensor data, which cannot be saved in on-chip memory, necessitating frequent transfers between on-chip and off-chip memory, and existing compression algorithms are inefficient without data similarity.
Innovation Solution
The solution involves sorting feature maps based on similarity during the training stage, using techniques like delta-based or dictionary-based compression, and storing them in formats like NHWC, where co-located elements are adjacent, reducing data redundancy and improving compression efficiency during the inference stage by applying sorted filters to input tensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If feature maps are stored in conventional memory formats without sorting, then memory storage is simpler, but compression efficiency is poor and memory bandwidth is wasted
Solution Approach 1:
The patent applies feature map sorting during the training stage, which is a preliminary action performed before the inference stage. By pre-sorting feature maps based on their spatial characteristics and similarity metrics, the system prepares the data in advance to enable more efficient compression during inference, reducing memory bandwidth requirements without adding complexity during the critical inference phase
Solution Approach 2:
The patent changes the organizational parameters of feature maps by sorting them according to spatial proximity and similarity metrics. This reorganization transforms the memory layout from a conventional format to one where similar features are grouped together, enabling delta-based compression algorithms to achieve better compression ratios by exploiting the increased data redundancy in the sorted arrangement
2Quantity of substance
If large activation tensor data is stored in off-chip memory, then memory capacity requirements are met, but memory transfer bandwidth is insufficient and power consumption increases
Solution Approach 1:
The patent changes the organizational parameters of activation tensor data by sorting feature maps based on spatial proximity and similarity. This reorganization enables more effective delta-based compression, reducing the amount of data that needs to be transferred between off-chip and on-chip memory, thereby reducing power consumption while maintaining the same memory capacity
Solution Approach 2:
The patent extracts and removes redundant information from activation tensor data through compression algorithms. By identifying and eliminating redundant data patterns in the sorted feature maps, the system reduces the volume of data that must be transferred across the memory interface, directly reducing power consumption associated with memory transfers
3Productivity
If feature maps are sorted and compressed during training stage, then inference stage memory transfers are reduced, but training stage processing time increases
Solution Approach 1:
The patent performs feature map sorting and compression setup during the training stage as a preliminary action. Although this requires additional processing time during training, it establishes an optimized data organization that significantly improves inference performance by reducing memory bandwidth requirements and enabling more efficient compression during the inference stage
Solution Approach 2:
The system uses the training data itself to determine the optimal sorting order of feature maps. By analyzing the training data's spatial characteristics and similarity metrics, the system automatically establishes a sorting scheme that is tailored to the specific model and dataset, making the training process serve the inference optimization goal
Data Source
AI summary
A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a similarity of the feature maps relative to each other and store the plurality of different feature maps in the memory.