Local Entropy Encoding for Block-Based Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression systems are inefficient in reducing memory requirements and communication bandwidth without compromising data quality and authenticity.
Innovation Solution
The encoding system generates a compressed representation of data by partitioning it into code symbol subsets and entropy encoding each subset using a dictionary of code symbol probability distributions or custom distributions learned from the data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data compression systems are used, then data can be stored and transmitted, but memory requirements and communication bandwidth are not reduced efficiently
Solution Approach 1:
The data stream is divided into multiple blocks, and each block is processed independently using separate probability distributions. This segmentation allows the system to adapt to local statistical variations in different parts of the data, improving compression efficiency while reducing the memory buffer required to maintain context across the entire data stream.
Solution Approach 2:
The system uses context-adaptive probability distributions that are specific to each block of data rather than a single global distribution. By learning and applying local statistical characteristics for each block, the encoder achieves better compression ratios without requiring large memory buffers to capture long-range dependencies.
2Quantity of substance
If conventional data compression systems are used, then data can be stored and transmitted, but communication bandwidth is not reduced efficiently
Solution Approach 1:
By processing data in blocks with independent probability models, the system achieves efficient compression that reduces the number of bits required for transmission. The block-based approach captures local redundancies effectively, minimizing the communication bandwidth needed while maintaining compression performance.
3Productivity
If compression is increased to reduce memory and bandwidth, then efficiency improves, but data quality and authenticity may be compromised
Solution Approach 1:
The system applies compression selectively at the block level, using full entropy coding where appropriate while maintaining the ability to preserve exact data representation when needed. This partial application of aggressive compression techniques maintains data authenticity while achieving high overall compression efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for compressing and decompressing data. In one aspect, an encoder neural network processes data to generate an output including a representation of the data as an ordered collection of code symbols. The ordered collection of code symbols is entropy encoded using one or more code symbol probability distributions. A compressed representation of the data is determined based on the entropy encoded representation of the collection of code symbols and data indicating the code symbol probability distributions used to entropy encode the collection of code symbols. In another aspect, a compressed representation of the data is decoded to determine the collection of code symbols representing the data. A reconstruction of the data is determined by processing the collection of code symbols by a decoder neural network.