Sparse Weight Tensor Compression With In-Line DMA Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large weight tensors in machine learning models exceed the memory capacity of computing devices, leading to increased latency and reduced throughput due to the need for distributed computations and significant memory bandwidth usage, even when sparse and requiring complex software management.
Innovation Solution
A compression technique that removes zero values from weight tensors and quantizes weight values to reduce storage size, using a direct memory access engine with an in-line decompression unit to decompress tensors on-the-fly, allowing parallel computation with the processing unit.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If weight tensors are stored in full precision without compression, then computation accuracy is maintained, but storage size and memory bandwidth usage increase significantly
Solution Approach 1:
The weight tensor is segmented into two separate storage structures: a first storage structure holds the primary weight values in full precision, while a second storage structure holds the delta values in reduced precision. This segmentation allows the system to maintain high computation accuracy for the most important values while reducing overall storage requirements for less critical adjustments.
Solution Approach 2:
The patent applies different precision parameters to different portions of the weight tensor. The primary weights are stored in full precision (e.g., 32-bit floating point), while the delta values are stored in reduced precision (e.g., 8-bit integers or lower). This parameter change strategy maintains accuracy where needed while reducing storage size for incremental updates.
2Quantity of substance
If weight tensors are compressed to reduce storage size, then memory bandwidth usage decreases, but decompression time and computational overhead increase
Solution Approach 1:
The delta values are pre-computed and stored in a compressed format during the training or weight update process. This preliminary compression action ensures that when the weights are loaded for inference or training, the decompression operation is minimized to a simple addition of the pre-compressed delta values to the base weights, rather than performing complex decompression algorithms during runtime.
3Quantity of substance
If distributed computations are used to handle large weight tensors, then memory capacity limitations are overcome, but system complexity and communication overhead increase
Solution Approach 1:
The patent extracts the incremental weight updates (delta values) from the full weight tensor and stores them separately in a compressed format. This extraction allows the system to work with a compact representation of weight changes rather than distributing and managing large portions of the full weight tensor across multiple devices, thereby reducing system complexity while still handling large model sizes.
4Measurement precision
If full precision weight values are loaded from memory, then computation accuracy is maintained, but memory access latency increases
Solution Approach 1:
The system changes the precision parameter based on the operational context. During training or fine-tuning, full precision weights are loaded to maintain accuracy. During inference or when applying pre-computed updates, the system uses the compressed delta values with reduced precision, thereby reducing memory access latency while maintaining sufficient accuracy for the task at hand.
Data Source
AI summary
To reduce the storage size of weight tensors and speed up loading of weight tensors from system memory, a compression technique can be employed to remove zero values from a weight tensor before storing the weight tensor in system memory. A sparsity threshold can be enforced to achieve a compression ratio target by forcing small weight values to zero during training. When the weight tensor is loaded from system memory, a direct memory access (DMA) engine with an in-line decompression unit can decompress the weight tensor on-the-fly. By performing the decompression in the DMA engine, expansion of the weight values back to the original weight tensor size can be carried out in parallel while other neural network computations are being performed by the processing unit.