Cluster Filesystem Synchronization via Consistency Points
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In scalable and writable caching systems, network connectivity issues between source and target sites can cause disruptions in synchronizing data and metadata updates, leading to inconsistencies and challenges in maintaining data availability and integrity across filesystems.
Innovation Solution
The method involves creating consistency points at specific points-in-time in the source filesystem, comparing these points to detect changes, and establishing links to update the target filesystem based on identified differences, ensuring data and metadata consistency across sites even during temporary disconnects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If data and metadata updates are continuously synchronized between source and target sites, then data consistency is improved, but network connectivity requirements increase and system reliability deteriorates during network failures
Solution Approach 1:
The system creates consistency points in advance at the source filesystem before network failures occur. These consistency points capture the state of data and metadata at specific moments, allowing the target site to recover and synchronize without requiring continuous network connectivity. This preliminary action ensures that when network failures happen, the system can resume synchronization from the last known consistent state rather than requiring constant connection.
Solution Approach 2:
Consistency points act as an intermediary mechanism between the source and target sites. Instead of requiring direct continuous communication, the consistency points serve as intermediate artifacts that store the state information, enabling asynchronous synchronization and allowing the system to tolerate network failures while maintaining eventual data consistency.
2Measurement precision
If consistency points are created frequently to track changes, then change detection accuracy is improved, but system overhead and complexity increase
Solution Approach 1:
The synchronization mechanism is segmented into discrete consistency points rather than attempting continuous monitoring. Each consistency point represents a specific snapshot in time, dividing the continuous change tracking problem into manageable discrete units. This segmentation allows the system to detect changes by comparing sequential consistency points without requiring complex continuous monitoring infrastructure.
Solution Approach 2:
Instead of implementing complex real-time change detection mechanisms, the system creates simplified copies of the filesystem state at periodic intervals (consistency points). These copies capture the essential state information needed for change detection, reducing the complexity of the synchronization mechanism while maintaining adequate change detection accuracy by comparing successive copies.
Data Source
AI summary
The embodiments described herein relate to synchronization of data in a shared pool of configurable computer resources. One or more consistency points are created in a source filesystem. A first consistency point is compared with a second consistency point to detect a directory change at the source filesystem, which includes identifying at least one difference between the first and second consistency points. A file level change associated with an established directory at a target filesystem is identified responsive to the detection of the directory change. A link is established between the source filesystem and the target filesystem, and the established directory is updated based on the file level change.