Delta Compression Candidate Selection Using Sketch Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current delta compression methods in data storage systems are inefficient in managing data chunks, leading to increased storage and bandwidth requirements due to the need to store and transmit entire data chunks, despite minor alterations, which results in redundant data storage and transmission.
Innovation Solution
Implementing a delta compression system that divides data into chunks, generates resemblance hashes (sketches) to identify similar chunks, and stores only the differences (delta) between chunks, along with metadata for reconstruction, using a preferential selection module to optimize base chunk selection based on similarity and access times, and incorporating deduplication and prefetching processes to reduce storage and bandwidth needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data chunks are stored and transmitted, then data integrity is maintained, but storage and bandwidth requirements increase significantly
Solution Approach 1:
The patent segments data into fixed-size chunks and further divides each chunk into variable-length sub-chunks based on content boundaries. This segmentation enables selective processing where only modified sub-chunks are delta-compressed rather than processing entire chunks, thus maintaining data integrity while reducing storage requirements.
Solution Approach 2:
The patent extracts only the modified portions (sub-chunks) from complete data chunks for delta compression processing. By identifying and extracting only the changed sub-chunks through fingerprint comparison, the system avoids storing redundant data while preserving the ability to reconstruct complete data integrity when needed.
2Quantity of substance
If delta compression is applied to all data chunks, then storage efficiency improves, but processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by performing delta compression only on identified sub-chunks that have changed, rather than processing entire data chunks. The system generates fingerprints for sub-chunks and applies compression selectively, reducing computational overhead while maintaining storage efficiency benefits where applicable.
Solution Approach 2:
By segmenting data chunks into sub-chunks, the system can process only the necessary portions that require compression. This segmentation reduces the scope of processing from entire chunks to only modified sub-chunks, thereby decreasing processing time and computational resource requirements.
3Device complexity
If base chunks are selected without optimization, then simplicity is maintained, but compression effectiveness decreases
Solution Approach 1:
The patent implements feedback mechanisms where the system monitors compression ratios and similarity metrics between data chunks. Based on this feedback, the preferential selection module dynamically adjusts base chunk selection criteria, choosing bases that maximize compression effectiveness while maintaining manageable process complexity through learned patterns.
Solution Approach 2:
The system changes parameters such as similarity thresholds and base chunk selection criteria based on data characteristics. The preferential selection module adjusts these parameters dynamically to optimize compression effectiveness for different data types and change patterns, balancing complexity and effectiveness.
Data Source
AI summary
A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and generates a sketch for the selected data chunk. The method and system search for a set of candidate data chunks with a matching sketch and rank the set of candidate data chunks by degree of sketch matching. The set of candidate data chunks are tie-braked using location status data for each candidate and the selected data chunk is delta compressed with a selected candidate data chunk. The delta compressed selected data chunk is then stored in a data storage system.