Variable Compression Units Entropy Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in data compression, particularly in determining when to combine data chunks for compression, leading to suboptimal storage savings and compression ratios.
Innovation Solution
A method is introduced that forms compression units by evaluating the entropy values of data chunks and determining whether to add new chunks based on estimated compression ratios and cumulative entropy values, ensuring that only chunks with entropy values below a threshold and similar entropy values are combined for compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data chunks are combined into compression units for compression, then storage savings improve, but compression ratio may deteriorate when chunks with different entropy values are combined
Solution Approach 1:
The patent applies local quality by evaluating and selecting data chunks based on their individual entropy values, ensuring that only chunks with similar entropy characteristics are combined into the same compression unit. This localized selection criterion maintains uniform compression properties within each unit, optimizing both storage savings and compression ratio.
Solution Approach 2:
The patent implements dynamics by making the compression unit composition adaptive rather than fixed. The system dynamically evaluates entropy values of data chunks and adjusts which chunks are combined together based on their compressibility characteristics, allowing the compression strategy to adapt to varying data patterns.
2Productivity
If entropy evaluation and selective combining is performed, then compression efficiency improves, but processing complexity increases
Solution Approach 1:
The patent applies preliminary action by calculating entropy values for data chunks before combining them into compression units. This advance evaluation allows the system to make informed decisions about chunk combination, improving compression efficiency without requiring complex real-time adjustments during the compression process.
3Loss of substance
If variable compression unit sizes are used, then storage optimization improves, but system complexity increases
Solution Approach 1:
The patent implements parameter changes by varying the size of compression units based on the entropy characteristics of the data being compressed. Rather than using a fixed compression unit size, the system adjusts the number of data chunks included in each compression unit according to their compressibility, optimizing storage while managing complexity through entropy-based decision rules.
Data Source
AI summary
Techniques for data processing a data set may comprise: performing first processing that forms a first compression unit, wherein the first compression unit includes a data chunks including a first data chunk having a first entropy value less than an entropy threshold, the first processing including: receiving a second data chunk; determining, in accordance with criteria, whether to add the second data chunk to the first compression unit; and responsive to determining to add the second data chunk to the first compression unit, adding the second data chunk to the first compression unit; and compressing the first compression unit as a single compressible unit. The second chunk may be added if its entropy value is less than the entropy threshold and if entropy values of the first and second chunks are similar. The second chunk may be added if the resulting compression unit provides sufficient storage/compression benefit.