Neural Network Activation Compression for Sparse Non-Zero Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for compressing activation data from neural networks are inefficient, particularly for sparse data, as they either compromise accuracy with lossy compression or do not adequately leverage the sparsity of the data.
Innovation Solution
A method that groups data elements into quad-trees, uses state indicators to identify non-zero values, and encodes these values using delta encoding and Golomb-Rice codes, allowing for efficient compression and decompression while maintaining data accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If lossy compression is used to compress activation data, then storage requirements are reduced, but data accuracy is compromised
Solution Approach 1:
The patent extracts and separately encodes only the non-zero values from the activation data, rather than compressing all data uniformly. By identifying and isolating the significant non-zero elements, the method achieves compression while preserving the critical information needed for accurate neural network computation, thus avoiding the accuracy loss typical of lossy compression methods
Solution Approach 2:
The patent applies different encoding strategies to different portions of the data based on their characteristics. Non-zero values are encoded with precision-preserving methods (delta encoding followed by Golomb-Rice coding), while zero values are handled efficiently through sparsity exploitation. This localized quality approach ensures that compression is applied where appropriate without compromising the accuracy of critical data elements
2Quantity of substance
If conventional compression methods are used on sparse activation data, then compression is applied uniformly, but compression efficiency is insufficient
Solution Approach 1:
The patent segments the activation data processing into distinct stages: (1) identifying non-zero values, (2) applying delta encoding to capture value changes, and (3) applying Golomb-Rice coding for final compression. This segmentation allows each stage to optimize for its specific purpose, achieving superior overall compression efficiency compared to uniform compression methods that treat all data equally regardless of sparsity
Solution Approach 2:
The patent changes the encoding parameters dynamically based on the data characteristics. Delta encoding uses the difference between consecutive non-zero values as its parameter, which adapts to the local data structure. Golomb-Rice coding further adapts by using a divisor parameter that can be optimized for the specific distribution of delta-encoded values. These parameter changes enable the compression method to efficiently handle the sparse and variable nature of activation data
Data Source
Figure 1
Figure 2
Figure 3a
AI summary
A method for compressing activation data of a neural network to be written to a storage is provided. The activation data is formed into a plurality of groups and a state indicator indicates whether there are any data elements within each group that have a non-zero value. A second state indicator indicates, for groups having a non-zero value, whether sub-groups within the group contain a data element having a non-zero value. A sub-group state indicator indicates, for each sub-group having a non-zero value, which data elements within that sub-group have a non-zero value. Non-zero values of data elements in the activation data are encoded and a compressed data set is formed comprising the first state indicators, any second state indicators, any sub-group state indicators and the encoded non-zero values.