Hash Entropy Analysis for Compression vs Deduplication Decisions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage management systems face inefficiencies due to the high cost and limited efficiency of advanced memory technologies, necessitating complex methodologies like deduplication and compression to optimize storage, but these methods may not always yield the best results.
Innovation Solution
A computer-implemented method that calculates a distance-preserving hash and performs entropy analysis to determine whether to compress or deduplicate data, using predefined thresholds to decide between compression and deduplication based on hash entropy levels, ensuring the most efficient storage operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to all data portions, then storage space is reduced, but processing time increases and compression efficiency decreases for high-entropy data
Solution Approach 1:
The patent changes the parameter being measured from raw data to distance-preserving hash values. By calculating hash entropy instead of data entropy, the system can quickly assess compressibility potential without processing the actual data, thus reducing processing time while maintaining accurate compression decisions
Solution Approach 2:
The patent performs preliminary hash calculation and entropy analysis before deciding whether to compress data. This preliminary action filters out high-entropy data that would not benefit from compression, avoiding wasted processing time on data portions that cannot be efficiently compressed
2Quantity of substance
If deduplication is applied to all data portions, then storage efficiency improves, but computational overhead increases and duplicate detection accuracy decreases
Solution Approach 1:
The patent transforms the deduplication problem from comparing raw data portions to comparing distance-preserving hash values. This parameter transformation reduces computational overhead significantly while maintaining the ability to detect duplicates through hash entropy analysis and threshold comparison
Solution Approach 2:
The patent introduces distance-preserving hash values as an intermediary between the original data and the deduplication comparison process. This intermediary enables efficient duplicate detection by allowing rapid hash-based filtering before performing more expensive operations
3Measurement precision
If hash entropy calculation is performed on raw data, then accuracy is high, but processing speed decreases
Solution Approach 1:
The patent changes the input parameter for entropy calculation from raw data bytes to distance-preserving hash values. This transformation maintains the entropy measurement's ability to predict compressibility and deduplication potential while dramatically reducing processing speed requirements, as hash functions are computationally lighter than direct entropy analysis on large data portions
Data Source
AI summary
A method, computer program product, and computing system for receiving a candidate data portion; calculating a distance-preserving hash for the candidate data portion; and performing an entropy analysis on the distance-preserving hash to generate a hash entropy for the candidate data portion.