Incremental Backup Agent for Snapshot Change Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing NAS systems lack the functionality to efficiently identify differences between snapshots, making incremental backups inefficient due to numerous network round trips and processing time for metadata comparison.
Innovation Solution
Implementing an incremental backup agent that uses SMB and NFS Direct protocols to issue a single RPC call for metadata retrieval, and performs checksum comparisons in a DFS manner to identify changes, reducing network round trips and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional NAS systems are used for incremental backups, then data backup functionality is provided, but the system requires numerous network round trips and excessive processing time to identify differences between snapshots
Solution Approach 1:
The system pre-computes and stores checksums for all files and directories in advance. When performing incremental backups, it directly compares these pre-computed checksums between snapshots to identify changes, eliminating the need for time-consuming real-time scanning and processing during the backup operation itself.
Solution Approach 2:
The system divides the large-scale backup task into smaller manageable units by computing checksums for individual files and directories separately. This segmentation allows for efficient comparison at the directory level first, and only drills down to file level when necessary, significantly reducing the overall processing time and network round trips.
2Productivity
If traditional NAS systems are used for incremental backups, then data backup functionality is provided, but the system requires numerous network round trips to fetch metadata
Solution Approach 1:
The system merges multiple metadata fetching operations into a single comprehensive RPC call. Instead of making separate network requests for each directory or file metadata, it consolidates all metadata retrieval into one efficient call, dramatically reducing network round trips and improving network efficiency.
Solution Approach 2:
The system pre-fetches and caches all necessary metadata in advance using a single RPC call. This preliminary action ensures that all metadata is locally available when change detection is needed, eliminating the need for repeated network round trips during the incremental backup process.
3Reliability
If conventional backup methods are used without change detection, then complete data copying is performed, but the process is inefficient and time-consuming
Solution Approach 1:
The system enables self-service change detection by automatically computing and comparing checksums between snapshots. This automated self-service mechanism identifies exactly which files have changed, allowing the backup process to copy only the necessary data rather than performing inefficient complete data copying, thus improving backup speed while maintaining data protection reliability.
Data Source
AI summary
An incremental backup agent performs backup operations that synchronize database on client side to a server database. In one embodiment, such backup operations are incremental backups, where the agent may identify differences between the current directory with the latest backed up version. The agent may issue a direct RPC using SMB protocols or NFS protocols to fetch all entries of directories with metadata in a single RPC call, instead of issuing one call to fetch metadata for each directory entry. The agent may identify changes with efficiency by performing checksum changes in a DFS manner. Starting from a root directory, the agent may generate a checksum for each directory and compare the checksums on the client side with the retrieved fingerprints, and if the backup agent identifies that the fingerprints match, the backup agent may then go to a deeper level and compare the fingerprints for child directories.