Off-Chip ML Data Compression Using Arithmetic Coding and Range Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for machine learning networks are inefficient and costly, particularly for off-chip memory accesses, due to their reliance on high-cost operations and sequential processing, which do not align with the high-bandwidth and data-parallel nature of machine learning workloads.
Innovation Solution
A lossless off-chip data compression system using arithmetic coding that exploits non-uniform value distributions in machine learning data, employing encoders and decoders to encode and decode data as pairs of symbols and offsets, allowing for high concurrency and wide datawidth accesses without modifying the neural network model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If arithmetic coding is used for data compression, then compression ratio and energy efficiency are improved, but device complexity increases due to the need for encoders and decoders
Solution Approach 1:
The patent introduces an intermediary compression system with encoders and decoders that sit between the machine learning model and off-chip memory. This intermediary layer performs arithmetic coding to compress data before storage and decompresses it during access, reducing the actual data volume in memory while maintaining full precision for computations. The encoder compresses data using arithmetic coding algorithms, and the decoder reconstructs original values, effectively mediating between storage constraints and computational requirements.
Solution Approach 2:
The patent changes the parameter representation of data by encoding values as ranges and probabilities rather than fixed-bit representations. Arithmetic coding dynamically adjusts the number of bits used to represent each data point based on its probability distribution, allowing high-probability values to use fewer bits while maintaining lossless reconstruction. This parameter transformation enables variable-length encoding that adapts to the actual data distribution in machine learning workloads.
2Quantity of substance
If off-chip memory size is reduced, then cost and power consumption are decreased, but data capacity and bandwidth are limited
Solution Approach 1:
The patent extracts the compression and decompression functionality from the main machine learning computation path and implements it as a separate, dedicated compression system. This extracted compression layer operates independently to reduce data volume before it enters the energy-intensive off-chip memory, thereby separating the data reduction function from the computational workload and allowing memory size reduction without impacting core ML operations.
Solution Approach 2:
The patent performs data compression as a preliminary action before data is stored in or accessed from off-chip memory. By compressing data in advance during the encoding phase and decompressing it before computational use, the system prepares data in a space-efficient format that reduces the physical memory capacity required, while ensuring full-quality data is available when needed for machine learning computations.
3Productivity
If compression is applied to data streams, then memory bandwidth and effective capacity are improved, but processing time increases due to encoding and decoding operations
Solution Approach 1:
The patent segments the data stream processing into distinct encoding and decoding phases that can operate independently and in parallel with other system operations. The compression system is divided into separate encoder and decoder components that process data in manageable chunks, allowing other machine learning operations to proceed simultaneously without being blocked by sequential compression/decompression bottlenecks.
Solution Approach 2:
The patent maintains continuous useful action by ensuring that compression and decompression operations are seamlessly integrated into the data flow pipeline. Data is compressed continuously as it is prepared for storage and decompressed continuously as it is needed for computation, without interrupting the overall machine learning workload. This continuous operation minimizes idle time and keeps the system productive throughout the compression/decompression cycle.
Data Source
AI summary
There is provided a system and method for compression and decompression of a data stream used by machine learning networks. The method including: encoding each value in the data stream, including: determining a mapping to one of a plurality of non-overlapping ranges, each value encoded as a symbol representative of the range and a corresponding offset; and arithmetically coding the symbol using a probability count; storing a compressed data stream including the arithmetically coded symbols and the corresponding offsets; and decoding the compressed data stream with arithmetic decoding using the probability count, the arithmetic decoded symbols use the offset bits to arrive at a decoded data stream; and communicating the decoded data stream for use by the machine learning networks.