Sub-Chunk Fingerprint Deduplication for Delta Storage Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current delta compression methods in data storage systems are inefficient in managing data chunks, leading to increased storage and bandwidth requirements due to the need to store and transmit entire data chunks, especially in scenarios with minor alterations, where repetitive data is not effectively minimized.
Innovation Solution
Implementing a delta compression system that divides data into chunks, generates resemblance hashes (sketches) to identify similar data chunks, and stores only the differences (delta) along with a reference to the base chunk, utilizing a preferential selection module to optimize base chunk choice based on similarity and access times, and incorporating pre-fetching and deduplication processes to enhance efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data chunks are stored and transmitted, then data integrity is maintained, but storage and bandwidth requirements increase significantly
Solution Approach 1:
The patent segments data into fixed-size chunks and further divides each chunk into multiple sub-chunks. This segmentation enables fingerprint-based deduplication at the sub-chunk level, allowing the system to store only unique sub-chunks and reconstruct entire data chunks from these segments, thereby reducing storage requirements while maintaining data integrity through reference tracking.
Solution Approach 2:
The patent creates fingerprint copies (hash values) of data chunks and sub-chunks to identify duplicates. Instead of storing redundant entire chunks, the system stores unique sub-chunks and uses fingerprint matching to identify and reference duplicate content, significantly reducing storage and bandwidth requirements while preserving data integrity through accurate reconstruction.
2Quantity of substance
If data is divided into chunks for delta compression, then storage efficiency improves, but the complexity of managing chunk relationships increases
Solution Approach 1:
By segmenting chunks into smaller sub-chunks with individual fingerprints, the system simplifies the management of chunk relationships. Each sub-chunk can be independently tracked, stored, and referenced, reducing the complexity of managing large chunk structures while improving storage efficiency through finer-grained deduplication.
Solution Approach 2:
The patent introduces fingerprint hashes as intermediaries between data chunks and storage management. These fingerprints serve as mediators that enable automatic identification, matching, and tracking of unique and duplicate sub-chunks, significantly reducing the complexity of managing chunk relationships while improving storage efficiency.
3Quantity of substance
If sub-chunk deduplication is implemented, then storage requirements are reduced further, but the computational overhead for fingerprint generation and matching increases
Solution Approach 1:
Segmenting data into sub-chunks enables parallel fingerprint generation and matching operations, distributing the computational overhead across multiple smaller units. This segmentation allows for more efficient processing and reduces the peak computational load compared to processing entire chunks, while achieving greater storage reduction through sub-chunk level deduplication.
Data Source
AI summary
A computer-implemented method and system for deduplicating sub-chunks in a data storage system selects a data chunk to deduplicate and generates a sketch for the selected data chunk. A similar data chunk is searched for using the sketch. A set of fingerprints corresponding to sub-chunks of the similar data chunk is loaded. The set of fingerprints for the similar data chunk is compared to a set of fingerprints of the selected data chunk and the selected chunk is encoded as a set of references to identical sub-chunks of the similar data chunk and at least one unmatched sub-chunk.