Delta Compression Candidate Selection for Storage Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current delta compression methods in data storage systems are inefficient in managing data chunks, leading to increased storage and bandwidth requirements, as they fail to effectively identify and store only the differences between similar data chunks, and do not optimize for access times and compression levels.
Innovation Solution
A delta compression system that utilizes fingerprint and sketch metadata to identify similar data chunks, prioritizes base chunk selection based on similarity and location status, and employs pre-fetching and deduplication techniques to reduce storage and bandwidth needs by storing only deltas and references, along with garbage collection and sanitization processes to maintain efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional backup methods store complete data chunks, then data integrity is maintained, but storage space and bandwidth requirements increase significantly
Solution Approach 1:
The patent segments data into fixed-size chunks and further divides chunks into variable-size blocks, enabling granular comparison and compression. This segmentation allows the system to identify and store only the differences (deltas) between similar data portions while maintaining complete data reconstruction capability through the combination of base chunks and deltas.
Solution Approach 2:
The patent extracts and stores only the essential difference information (deltas) between similar data chunks rather than storing complete copies. By calculating block-level deltas and storing only the changes along with references to base chunks, the system significantly reduces storage requirements while preserving data integrity through the ability to reconstruct original data.
2Quantity of substance
If delta compression is applied to all data chunks, then storage efficiency improves, but processing time and computational resources increase
Solution Approach 1:
The patent pre-calculates and stores fingerprint metadata for each data chunk before the actual delta compression process. This preliminary action enables rapid identification of potential base chunks through fingerprint matching, avoiding the need for exhaustive comparisons during the compression phase and significantly reducing processing time.
Solution Approach 2:
The patent implements a two-stage comparison process where it first performs a quick fingerprint-based filter to identify candidate base chunks, then applies detailed block-level delta calculation only to promising candidates. This partial application of the full compression process reduces computational overhead while maintaining compression effectiveness.
3Measurement precision
If the system stores detailed metadata for all data chunks, then data retrieval accuracy improves, but memory usage and index size increase
Solution Approach 1:
The patent extracts and stores only the essential fingerprint metadata (hash values of fixed-size blocks) for each data chunk in the index, rather than storing complete chunk data or detailed structural information. This selective extraction provides sufficient precision for identifying similar chunks while keeping memory usage manageable.
Solution Approach 2:
The patent applies different levels of metadata detail to different data structures: fingerprints are stored in the main index for quick lookup, while more detailed block-level metadata is stored locally with each data chunk. This local quality approach optimizes both retrieval accuracy and memory efficiency by placing detailed information only where needed.
Data Source
AI summary
A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and selects a set of candidate data chunks using a first selection mechanism. Throughput or resource utilization is monitored. A change is made to a second selection mechanism that increases similarity of the set of candidates with the selected data chunk to improve compression in response to determining high resource availability or high throughput level. A change is made to a third selection mechanism that increases throughput of the delta compression process in response to determining low resources availability or low throughput.