Hash-Based Backup Deduplication Catalog Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional backup systems consume significant compute power, storage bandwidth, and storage capacity, leading to increased operational costs and reduced efficiency, necessitating an improved method for accelerated and memory-efficient backup of data.
Innovation Solution
A computer-implemented method for accelerated and memory-efficient similarity matching of backup data, involving the generation of hashing values for data zones, creation of storage structures with containers for deduplication, and insertion or deletion of zone identifiers based on hash value presence, optimizing deduplication processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If conventional backup systems perform deduplication operations, then data redundancy is reduced, but compute power consumption and processing time increase significantly
Solution Approach 1:
The patent segments the deduplication process into multiple stages: first computing hash values for data zones, then performing similarity matching on hash values rather than raw data, and finally performing detailed comparison only on candidate matches. This segmentation reduces compute power consumption by avoiding full data comparisons for all data blocks.
Solution Approach 2:
The patent performs preliminary hashing of data zones before the actual deduplication comparison. By pre-computing and storing hash values, the system can quickly identify potential duplicates without performing expensive full data comparisons, thereby improving processing speed while maintaining deduplication effectiveness.
2Measurement precision
If conventional backup systems perform comprehensive similarity matching, then deduplication accuracy is improved, but memory consumption increases by more than 75%
Solution Approach 1:
The patent extracts only the essential identifying features (hash values) from the complete data zones for storage in the catalog. By storing only hash values and metadata rather than complete data blocks, the system achieves 75%+ memory reduction while maintaining the ability to accurately identify duplicates through hash comparison.
Solution Approach 2:
The patent applies different quality levels to different parts of the deduplication process: full data is retained locally for restoration, while only compact hash representations are stored in the catalog for comparison purposes. This local quality differentiation enables accurate similarity matching with minimal memory consumption.
3Reliability
If conventional backup systems store complete data blocks for deduplication, then data integrity is maintained, but storage capacity consumption increases
Solution Approach 1:
The patent uses hash values as compact copies or representations of the original data zones. These hash copies serve as proxies for data integrity verification, allowing the system to maintain reliability through hash comparison while storing only minimal catalog information rather than complete data blocks.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, a system, and a computer program product for performing accelerated and memory efficient similarity matching. A data stream having a plurality of data zones is received. Each zone includes a zone identifier. A plurality of hashing values for each zone are generated. Each hashing value is generated based on a portion of a zone. A storage structure having a plurality of storage containers is generated. Each storage container stores one or more hashing values associated with each respective storage container and a plurality of zone identifiers referencing the associated hashing values. At least one storage container includes a listing of zone identifiers stored in each storage container. Using the storage structure, the received data stream is deduplicated.