Block Dictionary Encoding for Parallel Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques using dictionary encoding are computationally intensive and inefficient due to the handling of variable-length codes, which hinder parallel processing and lead to increased CPU overhead and memory requirements, especially when dealing with large datasets and multiple columns.
Innovation Solution
The method involves subdividing a dataset into blocks, determining frequency distributions within each block, removing rows with less frequent values to reduce code-word lengths, and using block-specific dictionaries for encoding, allowing for parallel processing and efficient memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional entropy coding with variable-length codes is used, then compression ratio is improved, but processing speed and parallelism are degraded
Solution Approach 1:
The data is divided into fixed-size blocks that are processed independently. Each block is encoded separately using its own dictionary, enabling parallel processing of multiple blocks without the need to parse variable-length code boundaries. This segmentation resolves the contradiction by maintaining compression efficiency through block-level optimization while enabling CPU-level parallelism for improved processing speed.
Solution Approach 2:
The patent uses dynamic block size selection and adaptive dictionary construction for each block based on data characteristics. By dynamically adjusting the encoding parameters for each block rather than using a single global encoding scheme, the system achieves better compression ratios while maintaining regular code structures that support parallel processing.
2Loss of substance
If variable-length codes are used for entropy compression, then compression efficiency is improved, but code parsing complexity and CPU overhead are increased
Solution Approach 1:
By segmenting the data into fixed-size blocks with fixed-length codes within each block, the patent eliminates the need for complex variable-length code parsing. The fixed block structure provides natural boundaries that simplify the decoding process, reducing CPU overhead while maintaining compression efficiency through block-level optimization.
Solution Approach 2:
The patent introduces fixed-length code words as an intermediary representation between the original data and the compressed output. These fixed-length codes serve as a simplified intermediate form that eliminates the parsing complexity of variable-length codes while still achieving compression through dictionary-based encoding and selective row removal.
3Loss of substance
If global dictionaries are used for encoding, then compression ratio is improved, but memory usage and processing time are increased
Solution Approach 1:
The patent divides the global dictionary into multiple block-specific dictionaries, each containing only the code words relevant to its corresponding data block. This segmentation reduces the memory footprint of each dictionary while maintaining compression efficiency by using locally optimized code sets. Processors can load smaller block-specific dictionaries into cache, reducing memory bandwidth requirements.
Solution Approach 2:
Each block is encoded with a dictionary tailored to its specific data characteristics rather than using a single global dictionary. This local optimization allows each block to use the most efficient code words for its particular data pattern, improving compression ratio while keeping individual dictionary sizes manageable and suitable for cache storage.
4Loss of substance
If row removal for frequency optimization is performed, then code-word length is reduced, but computational overhead is increased
Solution Approach 1:
The patent performs row removal and frequency optimization independently for each block rather than on the entire dataset. This segmentation limits the computational scope of the row removal algorithm to manageable block sizes, reducing the overall computational overhead while still achieving code-word length reduction through local frequency optimization.
Solution Approach 2:
The patent applies row removal partially by removing only the least frequent rows needed to achieve optimal code-word lengths, rather than removing all rows or using excessive removal. This partial action achieves the necessary compression optimization with minimal computational overhead by stopping the removal process once the optimal code length is reached.
Data Source
AI summary
Embodiments relate to data compression using dictionary encoding. An aspect includes subdividing a table of uncompressed data into a first block and a second block of complete rows. Another aspect includes determining information about a frequency of occurrence of different values for each column of the first block. Another aspect includes selecting a row of the first block to be removed out of the first block using frequency of occurrence-information. Another aspect includes removing the a row out of the first block to form an updated first block and determining information about a frequency of occurrence of different values for each column of the updated first block. Another aspect includes deriving a dictionary containing code-words for encoding the values of the updated first block. Another aspect includes encoding the values of the updated first block based on the code-words. Another aspect includes adding the removed row to the second block.