File System Change Tracking for Snapshot-Scale Delta Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file system replication methods are inefficient and lack scalability, particularly in the context of delta generation, which is proportional to snapshot size rather than the number of changes, and do not support seamless replication across multiple clients.
Innovation Solution
A change tracking and delta generation (CTDG) system that generates deltas based on the number of changes between snapshots, enabling or disabling change tracking, and supports multiple clients, using key-value pairs and data structures to optimize replication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional file system replication methods are used, then replication can be performed, but delta generation is proportional to snapshot size rather than number of changes, resulting in inefficiency
Solution Approach 1:
The patent extracts only the changed portions (deltas) between snapshots using change tracking data structures, rather than copying entire snapshots. Change tracking maintains key-value pairs that record only modified files and directories, allowing the replication system to generate deltas proportional to the number of changes rather than snapshot size, directly improving replication efficiency and reducing time loss.
Solution Approach 2:
The patent changes the parameter for measuring replication work from snapshot size to number of changes. By introducing change tracking that records modifications at the file and directory level, the system transforms the delta generation process to scale with the number of changed items rather than the total snapshot size, resolving the efficiency problem.
2Productivity
If change tracking is enabled for multiple clients, then replication can be optimized, but system complexity increases
Solution Approach 1:
The patent introduces change tracking data structures (key-value pairs storing file paths, directory entries, and modification metadata) as an intermediary between the file system and replication process. This intermediary layer captures changes efficiently and provides structured information to the replication system, enabling optimized multi-client replication without proportionally increasing overall system complexity.
Solution Approach 2:
The change tracking mechanism serves multiple functions: it tracks file changes, records directory modifications, supports multiple clients simultaneously, and provides data for delta generation. This multi-functional approach allows a single system component to handle various replication requirements, reducing the need for separate mechanisms for each client and thereby limiting complexity growth.
3Reliability
If delta generation is based on snapshot size, then all changes are captured, but the process is not scalable to large file systems
Solution Approach 1:
The patent segments the file system into individual files and directories, with change tracking recording modifications at this granular level. Instead of treating the entire snapshot as a single unit, the system divides change tracking into discrete key-value pairs for each modified file and directory, enabling scalable processing of large file systems while maintaining complete change capture accuracy.
Solution Approach 2:
The patent changes the scaling parameter from snapshot size to number of changes. By measuring replication workload based on the count of modified files and directories rather than total snapshot size, the system achieves scalability to large file systems while maintaining reliable capture of all changes through the change tracking mechanism.
Data Source
AI summary
A change tracking and delta generation (CTDG) system servicing multiple clients is disclosed for enabling the delta generation during replication to be proportional to the number of changes between two snapshots instead of the size of the snapshots. In some embodiments, CTDG can track changes to a file system, where change-tracking can be dynamically switched between two sub-CT modes, file level or directory level. Change-tracking (CT) keys may be generated accordingly and stored in a data structure to be used for delta generation during replications for clients. In some embodiments, change-tracking may be enabled or disabled for different clients. CTDG can coordinate the CT enablement/disablement to ensure replications can use the information in CT keys properly.