Interleaved Weight Compression for Low-Latency Neural Network Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks require significant system bandwidth to read and process large arrays of weights, which can be reduced by compressing and decompressing these weights efficiently to minimize hardware requirements and latency.
Innovation Solution
A method of data compression and decompression that interleaves compressed data, allowing for efficient decompression with reduced hardware complexity and power consumption, applicable to multi-dimensional arrays including neural network weights, image, and voice data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weights are stored in uncompressed form in memory, then decompression speed is fast, but system bandwidth usage and hardware requirements increase significantly
Solution Approach 1:
The weight array is divided into multiple blocks, with frequently accessed weights placed in smaller blocks and less frequently accessed weights in larger blocks. This segmentation allows the system to load only necessary weight blocks into high-speed memory, reducing overall bandwidth requirements while maintaining fast access for critical weights.
Solution Approach 2:
Weight blocks are pre-ordered and organized in memory based on their access frequency and temporal patterns before inference begins. This preliminary organization enables the decompression apparatus to efficiently load blocks in the optimal sequence, minimizing decompression latency without requiring full uncompressed weight storage.
2Speed
If all weights are loaded into high-speed memory, then access speed is fast, but hardware complexity and power consumption increase
Solution Approach 1:
Different portions of the weight array are stored with different qualities - frequently accessed weights are kept in high-speed memory with full precision, while less frequently accessed weights are stored in slower memory with compressed or lower precision representation. This local differentiation optimizes the balance between access speed and hardware complexity.
Solution Approach 2:
The memory hierarchy is organized with small high-speed memory blocks nested within larger low-speed memory structures. The decompression apparatus loads only the necessary small blocks into high-speed memory during inference, avoiding the need to maintain all weights in high-speed memory simultaneously, thus reducing hardware complexity.
3Quantity of substance
If compressed weight blocks are stored, then memory usage is reduced, but decompression complexity increases
Solution Approach 1:
The compression scheme uses variable block sizes and adjustable compression ratios that can be configured based on the specific neural network layer and access patterns. This parameter flexibility allows optimization of the balance between compression ratio and decompression complexity for different weight blocks.
Solution Approach 2:
The decompression apparatus dynamically adjusts which weight blocks to decompress and to what precision based on real-time access patterns and layer requirements. This dynamic behavior allows the system to minimize decompression complexity by only decompressing necessary blocks at appropriate precision levels.
Data Source
Figure 1A~1B
Figure 2
Figure 3
AI summary
Methods of data compression and decompression are described. The compression method comprises encoding groups of data items by generating, for each group, header data comprising h-bits and a plurality of body portions each comprising b-bits and each body portion corresponding to a data item in the group. The value of h may be fixed for all groups and the value of b is fixed within a group, wherein the header data for a group comprises an indication of b for the body portions of that group. In various examples, b=0 and so there are no body portions. In examples where b is not equal to zero, a body data field is generated for each group by interleaving bits from the body portions corresponding to data items in the group. The resultant encoded data block, comprising the header data and, where present, the body data field can be written to memory.