Average Entropy Calculation for Storage Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern storage systems face inefficiencies due to the high cost and increasing speed of memory technology, necessitating complex methodologies like deduplication and compression to maximize storage efficiency.
Innovation Solution
A computer-implemented method that processes data by dividing it into chunks, performing entropy analysis, and determining average chunk entropy to decide on compression based on a predefined threshold, optimizing storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed using traditional entropy-based methods, then storage efficiency is improved, but the computational complexity and processing time increase significantly
Solution Approach 1:
The patent divides data into fixed-size chunks and calculates entropy independently for each chunk rather than processing the entire data set at once. This segmentation approach reduces the computational complexity of entropy calculation while maintaining storage efficiency, as each chunk can be processed separately and in parallel.
Solution Approach 2:
The patent performs entropy analysis on data chunks before the actual compression process to determine which chunks are suitable for compression. By pre-calculating entropy values and comparing them against thresholds, the system identifies compressible data portions in advance, avoiding unnecessary computational overhead during the compression phase itself.
2Speed
If advanced memory technology is used to increase speed, then processing performance is improved, but the cost of the storage system increases
Solution Approach 1:
The patent changes the parameter of data granularity by processing data in fixed-size chunks rather than as continuous streams. This parameter change enables more efficient utilization of memory systems by creating uniform data units that can be processed in batches, improving speed without requiring expensive high-performance memory technology.
Solution Approach 2:
The patent creates entropy value representations of data chunks without necessarily duplicating the actual data. By working with these compact entropy representations for comparison and decision-making, the system achieves fast processing using standard memory technology rather than requiring expensive high-speed memory to handle full data copies.
3Measurement precision
If data is divided into smaller chunks for entropy analysis, then compression accuracy is improved, but the overhead of processing increases
Solution Approach 1:
The patent implements a dynamic approach where the chunk size and entropy thresholds can be adjusted based on data characteristics and system conditions. This allows the system to optimize the balance between compression accuracy and processing overhead, using smaller chunks when accuracy is paramount and larger chunks when processing speed is more critical.
Solution Approach 2:
The patent employs feedback mechanisms where entropy calculation results from previous chunks inform the processing of subsequent chunks. By using the entropy values and compression outcomes of earlier chunks to guide subsequent processing decisions, the system reduces redundant calculations and minimizes overall processing overhead while maintaining high compression accuracy.
Data Source
AI summary
A method, computer program product, and computing system for processing a data portion to divide the data portion into a plurality of data chunks; performing an entropy analysis on each of the plurality of data chunks to generate a plurality of data chunk entropies; and determining an average data chunk entropy from the plurality of data chunk entropies.