Source Classification for Backup Deduplication via Fingerprint Cache
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current centralized backup systems using deduplication require significant computing and network resources due to the transmission of identifying information and responses between backup servers and central repositories, necessitating a reduction in these resource utilisations.
Innovation Solution
Implementing a fingerprint cache system where source data fingerprints are compared to a cache of previously backed-up fingerprints, reducing the need for queries and responses by only sending data segments not already stored in the central repository, thereby minimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication is implemented by transmitting identifying information for each data segment to the central repository, then data transmission volume is reduced, but computing and network resources are significantly consumed
Solution Approach 1:
The system performs preliminary actions by generating fingerprints of data segments before transmission and comparing them against a cache of previously seen fingerprints. This preliminary comparison allows the system to identify duplicate data segments locally without requiring network communication for each segment, thereby reducing both data transmission volume and the computing/network resources needed for actual deduplication operations
2Reliability
If the central repository processes and responds to each data segment query, then accurate deduplication is achieved, but processing time and computational load increase
Solution Approach 1:
The central repository performs preliminary actions by providing the backup server with a cache of fingerprints from previously backed-up data segments. This allows the backup server to perform preliminary local comparisons against this cache, filtering out obvious duplicates before any potential repository queries are needed, thereby maintaining deduplication accuracy while significantly reducing processing time
Solution Approach 2:
The fingerprint cache acts as an intermediary between the backup server and the central repository. Instead of the backup server directly querying the repository for every data segment, the fingerprint cache serves as a local intermediary that can quickly answer duplicate checks for commonly backed-up segments, reducing the need for time-consuming repository processing while maintaining accuracy
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system, method, and computer program product for backing up data from a backup source to a central repository using deduplication, where the data comprises source data segments is disclosed. A fingerprint cache comprising fingerprints of data segments stored in the central repository is received, where the data segments were previously backed up from the backup source. Source data fingerprints comprising fingerprints (e.g., hash values) of the source data segments are generated. The source data fingerprints are compared to the fingerprints in the fingerprint cache. The source data segments corresponding to fingerprints not in the fingerprint cache may not be currently stored in the central repository. After further queries to the central repository, one or more of the source data segments are sent to the central repository for storage responsive to comparison.