Delta Compression Garbage Collection for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current delta compression methods in data storage systems are inefficient in managing data chunks, leading to increased storage and bandwidth requirements due to the need to store and transmit entire data chunks, especially in scenarios with minor alterations or duplicates.
Innovation Solution
Implement a delta compression system that divides data into chunks, generates fingerprints and sketches for similarity matching, and stores only the delta between similar chunks, along with a reference to the base chunk, using modules for garbage collection, deduplication, pre-fetching, and preferential selection to optimize storage and transmission efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If delta compression is implemented to store only differences between data chunks, then storage requirements are reduced, but system complexity increases due to the need for fingerprint generation, sketch creation, and similarity matching mechanisms
Solution Approach 1:
The patent segments data into smaller units called data chunks, which are then processed individually through fingerprint generation and similarity matching. This segmentation enables selective storage of only the delta portions that differ between chunks, reducing overall storage requirements while managing complexity through modular processing
Solution Approach 2:
The system performs preliminary actions by generating fingerprints and sketches for data chunks before storage. These pre-computed identifiers enable rapid similarity matching and delta compression during subsequent operations, reducing the need for complex real-time analysis and lowering overall system complexity
2Reliability
If entire data chunks are stored for backup purposes, then data reliability is maintained, but bandwidth and storage resources are consumed unnecessarily for duplicate or minimally altered data
Solution Approach 1:
Instead of storing entire data chunks, the patent creates references or copies that point to base chunks. When data is retrieved, only the delta portions need to be transmitted and reassembled, maintaining data reliability while significantly reducing bandwidth and storage resource consumption for duplicate or minimally altered data
Solution Approach 2:
The system extracts only the essential difference information (delta) between data chunks and stores that instead of the complete data. This extraction approach maintains the ability to reconstruct original data for reliability while eliminating unnecessary storage and transmission of duplicate portions
3Productivity
If data is divided into smaller chunks for delta compression, then compression efficiency improves, but processing time and computational overhead increase due to increased number of operations
Solution Approach 1:
The patent performs fingerprint generation and sketch creation as preliminary actions during data ingestion. By pre-computing these identifiers, the system enables rapid similarity matching during compression operations, improving compression efficiency while reducing the computational overhead of repeated analysis during data processing
Solution Approach 2:
The system replaces complex mechanical data comparison operations with more efficient fingerprint-based identification and sketch matching mechanisms. This substitution enables faster determination of data similarity and reduces the computational time required for delta compression operations
Data Source
AI summary
A computer-implemented method and system for performing garbage collection in a delta compressed data storage system selects a file recipe to traverse to identify live data chunks and selects a chunk identifier from the file recipe. The chunk identifier is added to a set of live data chunks. Delta references in the file metadata corresponding to the chunk identifier are added to the set of live data chunks. Data chunks in a data storage system not identified by the set of live data chunks are then discarded.