Sparse Data Compression Using Unduplicated Values and Index Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Sparse data with redundant elements pose a challenge due to large overhead, as existing compression schemes are inefficient in handling and decompressing such data effectively.
Innovation Solution
A data compression method that generates compressed data with unduplicated values and index data indicating storage locations, allowing for lossless compression and decompression of sparse data, efficiently storing data in a smaller format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional sparse data compression schemes (e.g., CSR) are used, then data storage space is reduced, but decompression efficiency and data retrieval performance deteriorate
Solution Approach 1:
The patent segments the compressed sparse data into two separate components: (1) unduplicated value data stored in a compressed data structure, and (2) index data stored in a hash table that maps original positions to compressed positions. This segmentation allows independent optimization of storage and retrieval operations, enabling efficient decompression by directly looking up indices without sequential processing.
Solution Approach 2:
The patent introduces an intermediary hash table structure that acts as a mediator between the compressed unduplicated values and the original data positions. The hash table stores index pairs that map original data element positions to their corresponding positions in the compressed data, enabling O(1) direct access during decompression rather than requiring sequential traversal of compressed data.
2Ease of operation
If sparse data with redundant elements is stored without compression, then data retrieval simplicity is maintained, but storage overhead increases significantly
Solution Approach 1:
The patent extracts only the unique, non-redundant elements from the sparse data and stores them in a compressed structure, while separately maintaining index information that references these extracted elements. This extraction eliminates storage of duplicate zero or redundant elements, reducing storage overhead while preserving the ability to reconstruct the original data structure through the index mapping.
Solution Approach 2:
The patent changes the storage parameter from storing all original data elements (including duplicates) to storing only unduplicated elements with corresponding index metadata. This parameter change transforms the data representation from a dense format with redundancy to a compressed format that maintains retrieval capability through index-based access.
3Quantity of substance
If data is compressed to reduce size, then storage efficiency improves, but data integrity and lossless reconstruction may be compromised
Solution Approach 1:
The patent creates a compressed copy of the sparse data that preserves all necessary information for lossless reconstruction. By storing unduplicated values with index mappings that track their original positions, the system maintains a faithful representation of the original data structure, enabling exact reconstruction without loss of information or integrity.
Solution Approach 2:
The patent performs preliminary organization of data during the compression phase by identifying and categorizing unduplicated elements and their corresponding indices. This preliminary action ensures that all necessary information is properly structured and stored before decompression is needed, guaranteeing that data integrity is maintained and lossless reconstruction can be achieved when required.
Data Source
AI summary
One aspect of the present disclosure relates to a data compression method. The method includes generating, by one or more processors, compressed data from data, wherein the compressed data includes one or more unduplicated values of the data and generating, by the one or more processors, index data from the data, wherein the index data includes indices indicative of storage locations for the unduplicated values.