rANS Compression With Markov Probability Tables for Better Ratios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods, such as ANS, face challenges with static probability distributions, high computational costs, and inefficiencies in handling large datasets and inter-symbol redundancy, leading to suboptimal compression ratios and speed.
Innovation Solution
The use of Markov modeling to construct dynamic probability distribution tables for rANS encoding, allowing for iterative refinement and the incorporation of an escape code to handle unexpected symbols, along with SIMD processing for efficient lookup and compression of these tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If static probability distribution is used in rANS encoding, then encoding speed is improved, but compression ratio deteriorates
Solution Approach 1:
The patent applies dynamics by transitioning from static probability distributions to dynamic Markov models that adapt to local data patterns. The Markov models are constructed iteratively from subsets of data and updated as encoding progresses, allowing the probability distribution to evolve and capture inter-symbol dependencies while maintaining encoding speed through efficient table-based lookup structures.
2Loss of information
If probability distribution table is constructed on full dataset, then compression ratio is improved, but computational cost and time increase
Solution Approach 1:
The patent segments the full dataset into multiple subsets for iterative Markov model construction. Instead of building one large model on the entire dataset, the approach divides data into manageable portions, constructs models on each subset, and combines them progressively. This segmentation reduces memory requirements and computational overhead while maintaining compression effectiveness through the escape code mechanism.
Solution Approach 2:
The patent uses partial action by constructing Markov models on subsets rather than the complete dataset upfront. The iterative refinement process builds models progressively on portions of data, using escape codes to handle symbols not covered by current models. This approach achieves good compression ratios without the prohibitive cost of processing the entire dataset at once.
3Loss of information
If Markov model is used to capture inter-symbol redundancy, then compression ratio is improved, but device complexity increases
Solution Approach 1:
The patent implements feedback through iterative Markov model construction where the encoding process itself informs model refinement. As symbols are encoded, the Markov models are updated based on observed data patterns, creating a feedback loop that improves compression ratios by capturing inter-symbol dependencies. The escape code mechanism provides feedback when symbols fall outside current model predictions, triggering model updates.
4Reliability
If escape code is added to handle unexpected symbols, then reliability is improved, but encoding overhead increases
Solution Approach 1:
The patent applies beforehand cushioning by pre-allocating escape codes in the probability distribution tables before encoding begins. These escape codes serve as pre-prepared buffers to handle unexpected symbols that fall outside the current Markov model predictions. By having escape codes ready in advance, the system ensures reliable encoding of all possible symbols without corruption, while the escape code overhead is minimized through efficient table construction.
Data Source
AI summary
A data compression method using the range variant of asymmetric numeral systems to encode a data stream, where the probability distribution table is constructed using a Markov model. This type of encoding results in output that has higher compression ratios when compared to other compression algorithms and it performs particularly well with information that represents gene sequences or information related to gene sequences.