File Transfer Prioritization During Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication methods do not effectively prioritize files based on user or application criticality, leading to potential loss of critical files during replication, especially in disaster scenarios where only a few files are successfully transferred.
Innovation Solution
Implementing a system that prioritizes files and sub-paths based on user or application criticality by using a replication policy to identify priority sub-paths and files, ensuring that critical files are replicated first by tree-walking the priority sub-paths and adding them to a transfer queue before non-priority files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If files are replicated in arbitrary order without prioritization, then the replication process is simple to implement, but critical files may be lost during partial replication scenarios
Solution Approach 1:
The system performs preliminary actions by identifying and prioritizing critical files and sub-paths before the replication process begins. A replication policy is established in advance that designates certain files as critical based on user or application criticality. During replication, these pre-identified critical files are transferred first, ensuring that even if the replication must be interrupted, the most important data is protected. This preliminary prioritization resolves the contradiction by preparing the replication sequence beforehand, maintaining reliability without adding complex real-time decision-making during the transfer process.
2Reliability
If critical files are prioritized for replication first, then data protection is enhanced, but the replication process requires complex prioritization logic
Solution Approach 1:
The replication process is segmented into distinct phases: first, critical files identified by the replication policy are prioritized and transferred; second, remaining non-critical files are replicated. This segmentation creates a clear hierarchy in the replication queue, where critical files form a separate priority group. The complexity is managed by dividing the monolithic replication process into manageable segments with different handling rules, resolving the contradiction by organizing files into priority-based groups rather than treating all files uniformly.
Solution Approach 2:
Different quality of service is applied to different parts of the data set. Critical files, identified through the replication policy's designation of priority sub-paths, receive preferential treatment in the replication queue. Non-critical files follow standard replication procedures. This local differentiation allows the system to focus resources on protecting what matters most while maintaining simplicity for the bulk of data, resolving the contradiction by applying enhanced protection only where necessary rather than uniformly across all files.
3Reliability
If all files are added to the transfer queue simultaneously, then the replication process is straightforward, but critical files cannot be ensured to be transferred first
Solution Approach 1:
Before files are added to the transfer queue, the system performs a preliminary analysis using tree-walking to identify which files fall within priority sub-paths designated by the replication policy. This preliminary identification ensures that when the replication begins, the ordering is already determined. Critical files are added to the queue first based on this pre-established priority structure, guaranteeing they are transferred first without requiring complex real-time reordering during the replication process, thus maintaining both reliability and efficiency.
Data Source
AI summary
A system can to perform a replication of data from first computing equipment to second computing equipment, the data being stored as part of a path in a file system. The system can, from a replication policy for the replication, a priority sub-path of the path. The system can add a first set of files to a transfer queue comprising tree-walking the priority sub-path. The system can, after adding the first set of files, adding a second set of files to the transfer queue comprising tree-walking a portion of the path that is different from the priority sub-path. The system can replicate files in the transfer queue from the first computing equipment to the second computing equipment.