Sort-Based Symbol Index Encoding for Lossless Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques face limitations in handling various data types and achieving lossless reconstruction, especially for data with a high degree of sortedness, often requiring inefficient comparison cycles and not utilizing the distribution of data symbols effectively.
Innovation Solution
A sort-based binary encoder framework that sorts the indexes of data symbols and encodes the distribution information using a numerical binary encoder, reducing the number of comparisons by employing weights and selection bits, and utilizing a hybrid approach to minimize storage resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional comparison-based compression methods are used, then data can be compressed, but the number of comparisons required is large and the process is inefficient
Solution Approach 1:
The patent applies preliminary action by sorting the data symbols before compression. The sort-based encoder pre-organizes the input data in sorted order, which eliminates the need for repeated comparisons during the compression process. This preliminary sorting action transforms the data into a format that can be compressed more efficiently, reducing the time required during the actual compression operation.
Solution Approach 2:
The patent replaces the mechanical comparison-based compression mechanism with a sort-based encoding mechanism. Instead of repeatedly comparing data symbols during compression, the system uses the sorted structure of the data combined with numerical encoding to directly compute compression codes. This substitution of the underlying mechanism eliminates the inefficiency of repeated comparisons.
2Quantity of substance
If conventional compression methods are used, then compression is achieved, but storage efficiency is not optimized due to not utilizing data symbol distribution
Solution Approach 1:
The patent applies parameter changes by transforming the compression approach to utilize the distribution parameters of data symbols. The sort-based encoder analyzes the frequency and position distribution of symbols in the sorted data and adjusts the encoding parameters accordingly. This allows the compression scheme to adapt to the actual data characteristics, achieving better storage efficiency while preserving all necessary information for lossless reconstruction.
3Productivity
If sort-based encoding is applied, then compression efficiency improves, but the complexity of encoding distribution information increases
Solution Approach 1:
The patent applies universality by designing a numerical binary encoder that handles multiple encoding tasks through a unified framework. The same sort-based encoding mechanism works for different data types and distribution patterns, reducing the need for separate complex encoding logic for each case. This multi-functional approach simplifies the overall encoding complexity while maintaining high compression efficiency across various data scenarios.
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for data compression using a sort-based encoder are provided herein. An example method includes generating, for a sequence of data, a tree data structure comprising nodes including a set of leaf nodes. Each of at least a subset of the leaf nodes represents a corresponding data symbol and includes index values of instances of the corresponding data symbol in the sequence of data. The method includes applying a sort-based encoding process that merges the index values across the nodes. The sort-based encoding process includes merging the index values corresponding to a given set of the nodes and encoding selection information indicating which node the index value currently being merged belongs to and scaling information corresponding to weights maintained for the given set of nodes. A compressed version of the sequence of data is generated based on a result of the sort-based encoding process.