Parallel Change File Tracking for Distributed FSVM Backups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing backup technologies for distributed file server systems are inefficient in terms of time and resource utilization, particularly due to the need for full scans of changed files during incremental backups.
Innovation Solution
Implementing parallel change file tracking (CFT) on a distributed File Server Virtual Machine (FSVM) architecture, which allows for efficient incremental backups by identifying and backing up only changed bytes or blocks, and utilizing parallel threads to optimize network resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full scan for changed files is performed during incremental backup, then completeness of backup is ensured, but backup time increases significantly
Solution Approach 1:
The system performs preliminary actions by maintaining change file tracking metadata continuously updated in the background before backup operations are initiated. This pre-computation of change information eliminates the need for full scans during actual backup, ensuring completeness while reducing backup time.
Solution Approach 2:
The system creates and maintains copies of file metadata including change tracking information in dedicated data structures. These metadata copies allow the backup system to identify changed files without scanning the actual file contents, thus ensuring backup completeness while minimizing time consumption.
2Device complexity
If sequential backup processing is used, then system complexity is reduced, but network resource utilization is inefficient
Solution Approach 1:
The backup system segments the backup process into independent parallel tasks that can be executed simultaneously across multiple worker nodes. Each worker handles specific file or data stream independently, allowing concurrent network operations that maximize resource utilization while maintaining manageable system complexity through modular task design.
Solution Approach 2:
The system transitions from single-threaded sequential processing to multi-threaded parallel processing by adding the dimension of concurrency. Multiple backup operations execute simultaneously on different worker nodes, transforming the system from linear to parallel execution model, thereby optimizing network resource utilization without proportionally increasing complexity.
3Loss of time
If incremental backup of only changed files is implemented, then backup time is reduced, but the mechanism to track changes becomes more complex
Solution Approach 1:
The file system or storage layer automatically maintains change tracking metadata without requiring complex external monitoring systems. The change tracking mechanism serves itself by leveraging existing file system metadata structures and update operations, reducing backup time while keeping the tracking mechanism relatively simple through self-maintenance.
Data Source
AI summary
System and method for implementing parallel Change File Tracking (CFT) between a distributed File Server Virtual Machine (FSVM) system and a scale-out backup system using underlying snapshot technology. The FSVM system executes efficient assignments of backup streams between worker nodes of the backup system and computing nodes of the FSVM system based on the number of available worker nodes at the backup system. The assignment of backup streams allows for parallel processing of incremental backup data based on successive data file snapshots. The parallel incremental backup may, for example, be per node, per share, or per data size across nodes or shares.