Neural Network Activation Compression for Lower Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require high computational resources, making them unsuitable for long-duration operations in low-power devices like smartphones and AR/VR devices due to high power consumption, despite advancements in application-specific integrated circuits (ASICs), there is a need for improved performance and reduced power consumption in DNN processing.
Innovation Solution
A DNN module with a compression unit that reduces memory bus bandwidth by generating a mask portion and a data portion for compressed activation data, allowing for efficient storage and decompression, thereby reducing power consumption and enabling faster processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general-purpose processors like GPUs are used for DNN operations, then computational performance is improved, but power consumption increases making them unsuitable for long-duration operations in battery-powered devices
Solution Approach 1:
The patent segments the data transmission process into compression and decompression stages. The compression unit divides activation data into chunks and compresses them before storage, while the decompression unit reconstructs them during inference. This segmentation reduces memory bandwidth requirements and allows for more efficient power management in neural network operations.
Solution Approach 2:
The patent changes the parameter of data representation by transforming activation data from uncompressed format to compressed format. The compression unit applies compression algorithms that reduce the number of bits required to represent activation values, thereby reducing memory bandwidth utilization and power consumption during data transfer and storage operations.
2Speed
If memory bandwidth is increased to support high computational performance, then processing speed is improved, but power consumption increases
Solution Approach 1:
The patent segments activation data into manageable chunks that can be compressed and stored efficiently. By processing data in segmented blocks rather than continuous streams, the system reduces peak memory bandwidth requirements while maintaining processing throughput, thereby lowering power consumption during memory operations.
Solution Approach 2:
The compression unit performs preliminary compression of activation data before it is stored in memory or transmitted. This advance compression reduces the volume of data that needs to be transferred during inference operations, thereby reducing memory bandwidth utilization and power consumption without impacting processing speed.
3Quantity of substance
If compression is applied to activation data, then memory bandwidth utilization is reduced, but additional processing overhead is introduced
Solution Approach 1:
The compression and decompression units are integrated directly into the neural network processing pipeline, allowing them to operate autonomously on activation data as it flows through the system. This self-service approach minimizes external processing overhead by handling compression and decompression within the same processing domain, reducing the need for additional external processing resources.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
A deep neural network ("DNN") module can compress and decompress neuron-generated activation data to reduce the utilization of memory bus bandwidth. The compression unit can receive an uncompressed chunk of data generated by a neuron in the DNN module. The compression unit generates a mask portion and a data portion of a compressed output chunk. The mask portion encodes the presence and location of the zero and non-zero bytes in the uncompressed chunk of data. The data portion stores truncated non-zero bytes from the uncompressed chunk of data. A decompression unit can receive a compressed chunk of data from memory in the DNN processor or memory of an application host. The decompression unit decompresses the compressed chunk of data using the mask portion and the data portion. This can reduce memory bus utilization, allow a DNN module to complete processing operations more quickly, and reduce power consumption.