Sparse Data Compression Using Packed Indices and Value Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Sparse data with redundant elements pose a challenge due to large overhead, as existing compression schemes are inefficient in handling and storing such data effectively.
Innovation Solution
A data compression method that generates compressed data with unduplicated values and index data indicating storage locations, allowing for lossless compression and decompression of sparse data, efficiently storing data in a smaller format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sparse data is stored in conventional formats, then data integrity is maintained, but storage overhead becomes large
Solution Approach 1:
The patent segments sparse data into two separate components: compressed data (containing only non-zero values) and index data (containing positions and shape information). This segmentation allows efficient storage of only necessary data while maintaining the ability to reconstruct the original data structure, thereby reducing storage overhead without compromising data integrity.
Solution Approach 2:
The patent extracts and removes redundant zero elements from sparse data, storing only the non-zero values in compressed data. The index data stores the positional information needed to reconstruct the original data. This extraction approach eliminates storage overhead from redundant zeros while preserving complete data integrity through the index mapping.
2Quantity of substance
If data is compressed to reduce size, then storage efficiency improves, but decompression complexity increases
Solution Approach 1:
The patent performs preliminary organization during compression by creating structured index data that maps positions and shapes. This preliminary action stores the reconstruction roadmap in an organized manner, making the decompression process straightforward by simply following the pre-established index mappings rather than performing complex calculations.
Solution Approach 2:
The index data acts as an intermediary between the compressed data and the original data structure. It provides the necessary positional and shape information to correctly place compressed values in their original positions during decompression, simplifying the reconstruction process while maintaining data integrity.
3Quantity of substance
If conventional compression schemes are used for sparse data, then some compression is achieved, but handling efficiency remains poor
Solution Approach 1:
The patent applies different storage strategies to different parts of the data: non-zero values are stored in compressed format with full precision, while zero values are implicitly represented through the index structure. This local quality approach optimizes storage for non-zero elements while using the index to efficiently represent the sparse structure, improving both compression ratio and handling efficiency.
Solution Approach 2:
The patent transforms the storage problem from a single-dimensional array into a two-dimensional structure by separating data into compressed values and index information (positions and shapes). This dimensional change allows efficient handling of sparse data by operating on the index structure rather than the full data array, improving handling efficiency while maintaining compression.
Data Source
AI summary
A data processing method includes: acquiring, by one or more processors, compressed data generated from data, wherein values of the compressed data are stored at first storage locations, values of the data are stored at second storage locations; acquiring, by the one or more processors, index data includes indices indicative of the first storage locations; acquiring, by the one or more processors, at least two packed indices from the index data, the at least two packed indices being generated from the index data; and inputting, by the one or more processors, the at least two packed indices into at least two selectors.