Entropy-Weighted Similarity Digest for Data Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems struggle to effectively identify and reduce data redundancy due to the random nature of hash functions, which fail to distinguish between similar blocks with incompressible differences and dissimilar blocks with compressible differences, leading to suboptimal data reduction opportunities.
Innovation Solution
Generating entropy-weighted similarity digests that weight contributions from different regions of data chunks based on their entropy, allowing for the identification of similar chunks and performing data reduction by referencing one chunk to another, while storing differences separately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional hash functions are used for block matching, then identical blocks can be effectively identified, but similar blocks with incompressible differences cannot be distinguished from dissimilar blocks with compressible differences
Solution Approach 1:
The patent segments a data block into multiple sections (e.g., first section, second section, third section) and generates a separate digest for each section. This segmentation allows the system to analyze different parts of the block independently, identifying which sections contribute more to similarity and which contain compressible differences, thereby resolving the contradiction between accurate similarity identification and data reduction efficiency.
Solution Approach 2:
The patent applies different weighting factors to different sections based on their local characteristics. Sections with higher similarity weights are given more importance in the overall digest comparison, while sections with lower weights (indicating compressible differences) are de-emphasized. This local quality approach enables the system to distinguish between significant differences and compressible variations, improving both measurement precision and productivity.
2Productivity
If blocks are deduplicated based on overall similarity, then more deduplication opportunities are found, but residual differences may be incompressible and reduce storage efficiency
Solution Approach 1:
By segmenting blocks into sections and generating separate digests, the system can identify which sections contribute to similarity and which contain differences. This allows for targeted deduplication where only the compressible difference sections are stored separately, maximizing deduplication opportunities while maintaining storage efficiency by avoiding storage of incompressible differences.
Solution Approach 2:
The patent changes the parameter of digest generation by applying entropy-based weighting to different sections. This parameter change enables the system to prioritize sections with compressible differences for deduplication while identifying sections with incompressible differences that should be stored independently, thus improving both productivity and preventing loss of substance.
3Device complexity
If all sections of a block are weighted equally in digest generation, then the process is simple, but sections with compressible differences cannot be distinguished from sections with significant variations
Solution Approach 1:
The patent implements local quality by assigning different weights to different sections based on their entropy characteristics. Sections with lower entropy (indicating compressible differences) receive higher weights, while sections with higher entropy (indicating significant variations) receive lower weights. This approach improves measurement precision without excessive complexity, as the weighting can be computed efficiently from entropy measurements of each section.
Solution Approach 2:
The system changes the parameter of section weighting from uniform to entropy-based. By calculating entropy for each section and using it to determine weights, the system achieves better differentiation between compressible and incompressible differences. The complexity increase is manageable because entropy calculation is a standard operation and the weighted digest generation builds upon existing hash function capabilities.
Data Source
AI summary
A technique for managing data storage includes generating digests of chunks of data. The digests are arranged to provide similar results for similar chunks but to weight contributions from different regions of chunks differently based on their entropy. Chunks that differ in low-entropy regions tend to produce more similar digests than do chunks that differ in high-entropy regions. The technique further includes identifying similar chunks based on similarity of their digests and preforming data reduction on one chunk by reference to another, storing any differences between the chunks separately.