Off-Chip ML Data Compression Using Arithmetic Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for machine learning networks are inefficient in reducing off-chip memory access energy consumption, as they either target specific value patterns or impose additional constraints on neural network weights, while failing to effectively compress activation data and often require costly operations like division.
Innovation Solution
A lossless off-chip memory compression system using arithmetic coding that exploits non-uniform value distributions in machine learning workloads, encoding each value as a symbol and offset, and employing pipelining for high concurrency and bandwidth, compatible with various accelerators like SIMD units and systolic arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If existing data compression methods are used for machine learning networks, then some data compression is achieved, but they fail to effectively reduce off-chip memory access energy consumption and require costly operations like division
Solution Approach 1:
The patent extracts and removes the costly division operations from the compression algorithm by using alternative arithmetic operations (multiplication and addition) that are more efficiently implemented in hardware, thereby reducing computational complexity while maintaining compression effectiveness for machine learning workloads
Solution Approach 2:
The patent changes the operational parameters of the compression algorithm by replacing division-based methods with multiplication and addition-based methods, transforming the computational approach to better suit hardware implementation and reduce energy consumption in off-chip memory accesses
2Productivity
If data compression is applied at off-chip memory level, then effective capacity and bandwidth are boosted, but additional encoding and decoding overhead is introduced
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing probability count tables during an offline training phase, so that during actual inference no additional complex computations are needed at runtime, thereby reducing encoding and decoding latency while maintaining high compression ratios
Solution Approach 2:
The patent implements dynamic compression ratios by allowing the system to adaptively adjust the level of compression based on workload characteristics and performance requirements, optimizing the balance between bandwidth improvement and latency overhead
3Quantity of substance
If existing compression methods target specific value patterns, then those patterns are compressed effectively, but they impose additional constraints on neural network weights and fail to compress activation data
Solution Approach 1:
The patent creates a universal compression system that can handle both weight data and activation data without imposing constraints on neural network architectures, by using a probability-based approach that adapts to any data distribution rather than requiring specific value patterns
Data Source
AI summary
There is provided a system and method for compression and decompression of a data stream used by machine learning networks. The method including: encoding each value in the data stream, including: determining a mapping to one of a plurality of non-overlapping ranges, each value encoded as a symbol representative of the range and a corresponding offset; and arithmetically coding the symbol using a probability count; storing a compressed data stream including the arithmetically coded symbols and the corresponding offsets; and decoding the compressed data stream with arithmetic decoding using the probability count, the arithmetic decoded symbols use the offset bits to arrive at a decoded data stream; and communicating the decoded data stream for use by the machine learning networks.