Integer Compression Using Index-Offset Tuples for Sparse Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for random integers are inefficient, often requiring more computer memory than the original data and failing to achieve lossless compression, especially when dealing with large value universes or sparse values.
Innovation Solution
A data processing method that splits integers into index and data parts, generating block index and data tuples to store only the offset information, allowing for efficient storage and decompression of large sets of uniformly distributed integers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If known compression algorithms are used for random integers, then compression is achieved, but computer memory usage exceeds the raw size of the original data
Solution Approach 1:
The patent divides each integer into multiple parts (e.g., high-order bits and low-order bits) and stores them separately in different arrays. This segmentation allows the system to compress large integers by only storing the necessary portions, reducing overall memory usage while maintaining the ability to reconstruct the original values.
Solution Approach 2:
The patent extracts and stores only the essential components of the integer data. By identifying and storing only the unique or varying parts of the data (such as offsets or differences from a base value), the system reduces memory requirements while preserving the ability to recover the complete original integers when needed.
2Quantity of substance
If compression is applied to random data, then storage efficiency improves, but lossless compression becomes mathematically impossible
Solution Approach 1:
The patent transforms the representation parameters of the stored data by changing from storing complete integer values to storing segmented components or offset values. This parameter transformation enables compression of random data while maintaining lossless reconstruction capability, as the original values can be reliably recovered through mathematical operations on the compressed components.
3Adaptability or versatility
If large value universes or sparse values are compressed, then data coverage increases, but memory usage beyond raw size is required
Solution Approach 1:
The patent introduces a new dimensional approach by organizing compressed data into multiple arrays or dimensions (such as high-order bit arrays, low-order bit arrays, or offset arrays) rather than storing values in a single linear structure. This multi-dimensional organization allows efficient handling of large value universes and sparse data while reducing overall memory consumption through shared storage of common components.
Data Source
AI summary
A data processing method in the form of a data compression method is provided in which a plurality of integers are accessed. Each of the plurality of integers is split to generate a first plurality of numbers respectively paired with a second plurality of numbers. A first tuple is generated based on the first plurality of numbers. A second tuple is generated based on the second plurality of numbers and the first plurality of numbers. The first tuple and the second tuple are stored. A system and computer readable medium enabling the data processing method are further provided.