Delta Compression Candidate Ranking by Sketch Match and Location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current delta compression methods in data storage systems are inefficient in managing data chunks, leading to increased storage and bandwidth requirements due to the need to store and transmit entire data chunks, despite minor alterations, which can be effectively addressed by identifying and storing only the differences between similar data chunks.
Innovation Solution
Implement a delta compression system that divides data into chunks, generates fingerprints and sketches to identify similarities, and stores only the delta between chunks, along with metadata for reconstruction, utilizing modules for garbage collection, deduplication, pre-fetching, and preferential selection to optimize storage and transmission efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire data chunks are stored and transmitted, then data integrity is maintained, but storage and bandwidth requirements increase significantly
Solution Approach 1:
The patent segments data into fixed-size chunks and further divides each chunk into variable-size sub-chunks based on similarity boundaries. This segmentation allows the system to process and compress data at a finer granularity, storing only the necessary portions rather than entire chunks, thereby reducing storage requirements while maintaining data integrity through selective reconstruction.
Solution Approach 2:
The patent extracts and stores only the essential components for data reconstruction: sub-chunks representing actual data changes and references to base chunks. By taking out only the necessary elements (differences from base chunks) rather than storing complete data chunks, the system significantly reduces storage requirements while preserving the ability to reconstruct original data when needed.
2Loss of information
If entire data chunks are transmitted, then complete data is delivered, but bandwidth consumption increases
Solution Approach 1:
The patent segments data chunks into sub-chunks and transmits only those sub-chunks that represent actual changes or additions. By segmenting the transmission content to include only necessary data portions rather than complete chunks, the system reduces bandwidth consumption while ensuring data completeness through the reconstruction process that combines transmitted sub-chunks with base chunks.
Solution Approach 2:
The patent performs preliminary actions by pre-processing data into sub-chunks and identifying base chunks before transmission. This preliminary segmentation and organization allows the system to transmit only the minimal necessary data (sub-chunks representing changes) while maintaining the capability to reconstruct complete data at the destination, thereby reducing bandwidth consumption without sacrificing data completeness.
3Quantity of substance
If data is divided into smaller units for compression, then storage efficiency improves, but system complexity increases
Solution Approach 1:
The patent applies segmentation by dividing data chunks into sub-chunks based on similarity boundaries, which improves storage efficiency by enabling selective storage of only changed portions. The system manages this complexity through automated algorithms that identify similarity boundaries and organize sub-chunks with references to base chunks, making the complexity transparent to users while achieving significant storage efficiency gains.
Solution Approach 2:
The patent introduces an intermediary indexing structure that maps sub-chunks to their corresponding base chunks and stores metadata about data relationships. This intermediary layer manages the complexity of handling segmented data by providing an organized framework for storing, retrieving, and reconstructing data from sub-chunks, thereby improving storage efficiency while containing system complexity through structured organization.
4Quantity of substance
If only delta changes are stored, then storage requirements are reduced, but data reconstruction complexity increases
Solution Approach 1:
The patent extracts and stores only the essential elements for reconstruction: sub-chunks representing data changes and references to base chunks. By taking out only the necessary components rather than storing complete data, the system reduces storage requirements. The reconstruction complexity is managed through the organized structure of sub-chunk references and base chunk mappings, which provide a clear pathway for reconstructing original data when needed.
Solution Approach 2:
The patent uses copying by storing references to base chunks instead of duplicating entire data chunks. When reconstruction is needed, the system copies only the necessary sub-chunks and combines them with the referenced base chunks to recreate the original data. This copying approach reduces storage requirements while managing reconstruction complexity through efficient reference-based retrieval and combination operations.
Data Source
AI summary
A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and generates a sketch for the selected data chunk. The method and system search for a set of candidate data chunks with a matching sketch and rank the set of candidate data chunks by degree of sketch matching. The set of candidate data chunks are tie-braked using location status data for each candidate and the selected data chunk is delta compressed with a selected candidate data chunk. The delta compressed selected data chunk is then stored in a data storage system.