Neural Activation Compression Using Grouped Zero-State Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for compressing activation data in neural networks are inefficient, particularly due to the presence of a significant number of zero values, which can lead to suboptimal processing efficiency and potential loss of accuracy in lossy compression schemes.
Innovation Solution
A method involving grouping data elements into units, identifying non-zero values, and encoding them using delta encoding and Golomb-Rice codes, along with state indicators to form a compressed data set that includes first and second state indicators and sub-group state indicators, allowing for efficient compression and decompression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If activation data is compressed using conventional entropy coding algorithms, then data storage requirements are reduced, but processing efficiency deteriorates due to the presence of significant zero values
Solution Approach 1:
The patent segments activation data into multiple groups, where each group is processed independently to identify zero values. This segmentation allows the system to efficiently track and compress zero values without requiring complex processing of the entire activation map, thereby reducing storage requirements while maintaining processing efficiency.
Solution Approach 2:
The patent performs preliminary identification and marking of zero values before the main compression process. By pre-processing the data to identify zero positions and creating a compressed representation in advance, the system reduces the computational burden during actual compression and improves overall processing efficiency.
2Quantity of substance
If lossy compression schemes are applied to activation data, then compression ratio is improved, but accuracy deteriorates due to potential loss of important information
Solution Approach 1:
The patent changes the representation parameters of activation data by using a specialized format that distinguishes zero values from non-zero values. This parameter change allows for more efficient encoding where zero values can be represented with fewer bits, achieving better compression ratios while preserving all original information for accurate reconstruction.
3Device complexity
If conventional compression methods are used without special handling of zero values, then implementation complexity is reduced, but compression efficiency deteriorates due to suboptimal handling of sparse data
Solution Approach 1:
The patent divides the activation map into multiple groups and processes each group separately to identify zero values. This segmentation approach maintains relatively simple implementation within each group while achieving superior overall compression efficiency for sparse activation data.
Solution Approach 2:
The patent introduces an intermediary compressed format that serves as a bridge between the original activation data and the final stored representation. This intermediary format efficiently captures zero value positions and enables better compression without requiring complex direct transformation methods.
Data Source
AI summary
A method for compressing activation data of a neural network to be written to a storage is provided. The activation data is formed into a plurality of groups and a state indicator indicates whether there are any data elements within each group that have a non-zero value. A second state indicator indicates, for groups having a non-zero value, whether sub-groups within the group contain a data element having a non-zero value. A sub-group state indicator indicates, for each sub-group having a non-zero value, which data elements within that sub-group have a non-zero value. Non-zero values of data elements in the activation data are encoded and a compressed data set is formed comprising the first state indicators, any second state indicators, any sub-group state indicators and the encoded non-zero values.