Two-Level Fingerprint Caching for Backup Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems face inefficiencies in scalability, network resource utilization, and restore performance due to high query and response volumes between clients and deduplication servers, particularly exacerbated by complex data distribution and rapid data growth.
Innovation Solution
Implementing a two-level fingerprint caching scheme that pre-populates client caches with relevant fingerprints, reducing queries to the server cache by performing local queries and maintaining temporal and spatial locality through selective caching and periodic maintenance of server cache entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If all fingerprint queries are sent to the deduplication server, then centralized deduplication management is maintained, but network bandwidth consumption increases and server workload increases
Solution Approach 1:
The system pre-populates the client cache with fingerprints from previous backup images before the actual deduplication process begins. This preliminary action allows the client to perform local queries against cached fingerprints from prior backups, reducing the need to query the server for every fingerprint during the current backup operation.
Solution Approach 2:
The invention implements a two-level caching architecture where the client cache stores fingerprints locally at the client device, while the server cache maintains fingerprints on the deduplication server. This local quality enables the client to handle queries for previously backed-up data locally, reserving network bandwidth for new or uncached data.
2Speed
If fingerprints are cached at the client level, then query speed improves, but cache memory is consumed
Solution Approach 1:
Instead of caching all possible fingerprints, the system caches only a subset of fingerprints from previous backup images in the client cache. This partial caching approach balances memory consumption with query performance, storing enough historical fingerprints to provide benefit while limiting memory usage.
Solution Approach 2:
The caching system is segmented into two distinct levels: client cache and server cache. The client cache handles recent or frequently accessed fingerprints locally, while the server cache maintains a comprehensive fingerprint database. This segmentation allows the system to optimize for both speed and memory efficiency.
3Reliability
If the server maintains a comprehensive fingerprint database, then deduplication accuracy is improved, but server processing load increases
Solution Approach 1:
The client cache acts as an intermediary between the client and the server cache. It filters and pre-processes fingerprint queries by handling lookups for cached fingerprints locally, thereby reducing the number of queries that reach the server and lowering server processing load while maintaining deduplication accuracy.
Solution Approach 2:
The server pre-computes and stores fingerprints for all backup images in the server cache before client queries occur. This preliminary computation eliminates the need for real-time fingerprint calculation during backup operations, reducing server processing load during actual deduplication while maintaining comprehensive fingerprint coverage for accurate deduplication.
4Productivity
If client cache is pre-populated with historical fingerprints, then cache hit ratio improves, but initial setup time increases
Solution Approach 1:
The client cache is pre-populated periodically using incremental backup images rather than requiring a complete population from all historical backups at once. This periodic pre-population approach, triggered by incremental backups, gradually builds the cache over time, reducing the perceived setup time while still achieving high cache hit ratios.
Solution Approach 2:
The system performs preliminary population of the client cache using fingerprints from the most recent incremental backup image before the main deduplication operation begins. This preliminary action provides immediate benefit without requiring processing of the entire backup history, thus limiting setup time while improving cache effectiveness.
Data Source
AI summary
The present disclosure provides for implementing a two-level fingerprint caching scheme for a client cache and a server cache. The client cache hit ratio can be improved by pre-populating the client cache with fingerprints that are relevant to the client. Relevant fingerprints include fingerprints used during a recent time period (e.g., fingerprints of segments that are included in the last full backup image and any following incremental backup images created for the client after the last full backup image), and thus are referred to as fingerprints with good temporal locality. Relevant fingerprints also include fingerprints associated with a storage container that has good spatial locality, and thus are referred to as fingerprints with good spatial locality. A pre-set threshold established for the client cache (e.g., threshold Tc) is used to determine whether a storage container (and thus fingerprints associated with the storage container) has good spatial locality.