Cloud-Native File System Push Classes for Multi-Site Data Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-site shared volumes, there is a need for operators to determine when files have been successfully pushed from a source filer to target filers, ensuring all filers have a consistent view of new or modified data.
Innovation Solution
A mechanism is implemented to associate files into a 'push class' and initiate a push operation, which includes taking a snapshot of the volume. This operation pushes file data and metadata in phases, allowing the local filer to determine if the new version of the file is available at other filers by querying their metadata or payload identifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If files are pushed from source filer to target filers in a multi-site shared volume, then data consistency across filers is improved, but the complexity of determining when files have arrived at targets increases
Solution Approach 1:
The patent applies preliminary action by pushing metadata about upcoming file transfers before the actual file data is transferred. This allows target filers to prepare in advance and know what files to expect, eliminating the need for complex post-transfer verification. The metadata push occurs as a preliminary step that enables the main file transfer process to proceed efficiently with simpler status tracking.
Solution Approach 2:
The patent introduces metadata as an intermediary element that mediates between source and target filers during file transfers. This metadata acts as a signaling mechanism that carries information about file transfer status, timing, and expectations without requiring direct complex communication between filers. The metadata intermediary simplifies the determination of push status by providing structured information that can be easily parsed and acted upon.
2Reliability
If file data and metadata are pushed in separate phases, then data integrity is improved, but the time required for complete replication increases
Solution Approach 1:
The patent applies preliminary action by pushing metadata before file data in separate phases. The metadata phase prepares target filers with information about upcoming transfers, allowing them to allocate resources and prepare storage in advance. This preliminary metadata push enables the subsequent data phase to proceed more efficiently, reducing overall replication time while maintaining data integrity through the structured two-phase approach.
Solution Approach 2:
The patent segments the file transfer process into distinct phases: a metadata push phase and a data push phase. This segmentation allows each phase to be optimized independently - the metadata phase can be completed quickly with minimal overhead, while the data phase can proceed with full bandwidth utilization. The separation ensures data integrity by establishing a clear protocol structure while reducing total replication time through efficient phase management.
Data Source
AI summary
A technique for data sharing among multiple filers that share a volume in a private or public cloud object store is implemented. In this approach, a mechanism is provided to enable a local filer to determine whether other filers that are sharing the volume have a consistent view of new data being written to the cloud object store by the local filer. The begins by associating together a collection of one or more files in a “push class.” On demand, a push operation for the push class is initiated on the local filer. Preferably, the push is managed according to one or more push criteria associated with the push class. Typically, the push operation pushes file data and metadata associated with the one or more files of the push class in respective phases, with the file data being pushed to the cloud during a first phase and the metadata associated with that file data being pushed during a second phase that follows the first phase. After the push operation completes, a determination is made whether the new version of the file is available at one or more other filers that share the volume.