Live File System Data Replication via FID Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication systems require a snapshot of the file system during initial synchronization, which can be inefficient and may not account for changes occurring during the scanning process, leading to potential inconsistencies and increased data loss in case of system failures.
Innovation Solution
The system employs file identifier descriptors (FIDs) to scan and populate a file name database for a live file system, allowing for efficient data replication by tracking files and combining write operations, and excluding temporary files from replication to enhance data management and reduce data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a snapshot of the file system is used during initial synchronization, then data consistency is ensured, but data replication efficiency decreases and data loss increases
Solution Approach 1:
The patent transitions from static snapshot-based scanning to dynamic live file system scanning. The system continuously monitors and scans the live file system during replication, allowing it to adapt to changing file structures and capture real-time data, thereby eliminating the need for snapshots while maintaining consistency.
Solution Approach 2:
The patent implements continuous scanning of the live file system during the replication process. Instead of stopping to take a snapshot, the system maintains continuous monitoring and scanning operations, ensuring that data is consistently captured without interruption and improving overall replication efficiency.
2Reliability
If a snapshot of the file system is used during initial synchronization, then data consistency is ensured, but data loss increases
Solution Approach 1:
The system dynamically scans the live file system during replication, capturing data as it exists at the moment of scanning. This dynamic approach ensures that all current data, including changes made after snapshot creation, is included in replication, thereby preventing data loss while maintaining consistency.
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously monitors the live file system and adjusts its scanning and replication processes accordingly. This feedback loop ensures that any changes to the file system are detected and incorporated into the replication process, preventing data loss.
3Productivity
If FIDs are used to scan a live file system, then data replication efficiency improves, but system complexity increases
Solution Approach 1:
The patent introduces FIDs (File Identifier Descriptors) as an intermediary mechanism between the scanning process and the file system. FIDs serve as a simplified interface that allows the system to efficiently track and identify files during live scanning, reducing the complexity of direct file system interaction while improving replication efficiency.
4Quantity of substance
If temporary files are included in replication, then data completeness is maintained, but data management complexity increases
Solution Approach 1:
The patent extracts and separates temporary files from the main replication process. By identifying and excluding temporary files from replication, the system maintains data completeness for permanent files while simplifying data management by not duplicating transient data, thereby reducing overall complexity.
Data Source
AI summary
Preparing source data to be replicated in a continuous data replication environment. Certain systems and methods populate a file name database with entries having a unique file identifier descriptor (FID), short name and a FID of the parent directory of each directory or file on a source storage device. Such information is advantageously gathered during scanning of a live file system without requiring a snapshot of the source storage device. The database can be further used to generate absolute file names associated with data operations to be replayed on a destination storage device. Based on the obtained FIDs, certain embodiments can further combine write operations to be replayed on the destination storage device and/or avoid replicating temporary files to the destination system.