Delta Compression Candidate Selection for Storage Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in managing and compressing large amounts of data, particularly in backup systems, where repetitive or minor alterations lead to redundant data storage, and existing delta compression methods do not effectively optimize bandwidth usage during data transmission.
Innovation Solution
A delta compression system that divides data into chunks, generates fingerprints and sketches for similarity matching, and stores only the differences (delta) between similar chunks, along with metadata for reconstruction, utilizing garbage collection, deduplication, and pre-fetching processes to enhance storage efficiency and reduce bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data chunks are stored in their entirety, then data integrity and accessibility are ensured, but storage space and bandwidth requirements increase significantly
Solution Approach 1:
The system segments data into fixed-size chunks and processes each chunk independently through hashing and similarity comparison. This allows selective storage of only necessary data portions while maintaining the ability to reconstruct complete files through chunk assembly and delta application.
Solution Approach 2:
The system transforms data representation by storing hashes (32-byte fingerprints) and delta differences instead of complete chunk copies. This parameter change from storing full data to storing compressed representations significantly reduces storage requirements while preserving data integrity through reversible transformation.
2Quantity of substance
If delta compression is applied to all data chunks, then storage efficiency improves, but processing time and computational resources increase
Solution Approach 1:
The system pre-computes and stores hash fingerprints for all data chunks in an index before actual compression operations. This preliminary action allows rapid similarity detection through hash comparison rather than full data comparison, significantly reducing processing time during compression operations.
Solution Approach 2:
The system extracts only the essential identifying feature (hash fingerprint) from each data chunk for storage in the index, separating this identification function from the full data storage. This extraction enables efficient similarity matching without requiring access to or processing of complete chunk data during compression decisions.
3Manufacturing precision
If multiple candidate base chunks are evaluated for delta compression, then compression quality improves, but selection complexity and processing overhead increase
Solution Approach 1:
The system changes the selection criterion from evaluating multiple complex similarity metrics to a single hash comparison parameter. By transforming the base chunk selection into a simple hash match operation, the system maintains compression quality through accurate similarity detection while dramatically reducing selection complexity and processing overhead.
Solution Approach 2:
The system creates a simplified copy (hash fingerprint) of each base chunk's identifying characteristics and stores it in the index. This copy enables rapid comparison and selection without requiring access to or analysis of the actual chunk data, reducing selection complexity while maintaining compression quality through accurate similarity matching.
Data Source
AI summary
A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and selects a set of candidate data chunks using a first selection mechanism. Throughput or resource utilization is monitored. A change is made to a second selection mechanism that increases similarity of the set of candidates with the selected data chunk to improve compression in response to determining high resource availability or high throughput level. A change is made to a third selection mechanism that increases throughput of the delta compression process in response to determining low resources availability or low throughput.