Block-Based Data Encoding for High-Entropy Compression

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing lossless data compression algorithms, such as dictionary-based LZ and Huffman coding, face inefficiencies in compressing data with high entropy, particularly for continuous data with random patterns, resulting in lower compression ratios.

Innovation Solution

A data encoder that preprocesses data into sub-blocks, generates meta-data, and uses a Huffman algorithm to form tree structures and update codewords, effectively reducing entropy and improving compression ratios by dividing data into manageable groups and adjusting symbol frequencies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If Huffman coding is applied to continuous data with random patterns, then the encoding process can be performed, but the compression ratio decreases due to high entropy

Engineering Contradiction:

Improvecompression ratioVSAvoidentropy

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The patent divides continuous data into fixed-size blocks before applying Huffman coding. Each block is independently processed to create a block-specific frequency table and codeword assignment. This segmentation allows the encoder to capture local patterns in data that would be diluted in global statistics, thereby improving compression ratios for continuous data with random patterns while maintaining manageable computational complexity.

Inventive Principle:

Principle #1Segmentation

2Productivity

If dictionary-based LZ algorithm is used, then high compression ratio is achieved for data with repetitive patterns, but compression ratio is lowered for other data types

Engineering Contradiction:

Improvecompression ratioVSAvoiddata pattern adaptability

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent dynamically adjusts Huffman coding parameters (block size, frequency table construction method) based on the statistical characteristics of the input data. By analyzing data properties and adapting encoding parameters accordingly, the system achieves high compression ratios for both repetitive patterns and random data, overcoming the limitation of fixed-parameter LZ algorithms that perform poorly on non-repetitive data.

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If block Huffman coding is applied to each block, then continuous data can be processed, but the overall compression efficiency is reduced compared to full-data Huffman coding

Engineering Contradiction:

Improveprocessability of continuous dataVSAvoidcompression efficiency

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent performs preliminary statistical analysis on each data block to construct accurate frequency tables before encoding. By pre-processing each block to identify symbol frequencies and optimize codeword assignments based on local statistics, the system achieves compression efficiency close to full-data Huffman coding while maintaining the operational advantage of processing continuous data in manageable blocks.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10498358B2Data encoder and data encoding method

Publication Date: 2019.12.03 SK HYNIX INC

AI summary

A data encoder including a preprocessor configured to divide a data stream into a plurality of sub data blocks; a plurality of meta data generators each configured to generate meta data from one of the plurality of sub data blocks; and a plurality of data compressors each configured to compress one of the plurality of sub data blocks according to the meta data.