Hierarchical Cache Layout for Parallel File System Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file system cross-region replication technologies face inefficiencies and performance challenges due to dependency relationships between parent directories and child files, leading to increased IO traffic and latency, especially when storage devices are remote, and hinder parallel processing.
Innovation Solution
A hierarchical caching mechanism is implemented, utilizing a B-tree layout with three levels of caches (first-level, second-level, and third-level caches) to store and manage replication-related information efficiently, enabling parallel processing and resolving snapshot key inter-dependency by creating dummy entries to allow immediate processing of dependent information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional file system cross-region replication is used with dependency relationships between parent directories and child files, then data consistency is maintained, but IO traffic increases and latency increases
Solution Approach 1:
The patent segments the file system metadata into separate B-tree structures: one B-tree for parent directory information and separate B-trees for child files. This segmentation allows independent processing and caching of parent directory data without waiting for child file dependencies, reducing latency while maintaining data consistency through the hierarchical cache structure.
Solution Approach 2:
The patent implements preliminary action by creating dummy B-tree entries for parent directories before actual child file data is available. This allows the replication system to proceed with processing parent directory information in advance, reducing waiting time and latency while ensuring data consistency is maintained when the actual data arrives.
2Manufacturing precision
If traditional file system cross-region replication is used with dependency relationships, then data accuracy is preserved, but parallel processing is hindered
Solution Approach 1:
The patent divides the file system into independent segments: parent directory B-trees and child file B-trees are separated. This segmentation enables parallel processing across different segments while maintaining data accuracy through the hierarchical cache structure that ensures proper data relationships are preserved during replication.
Solution Approach 2:
The patent introduces dummy B-tree entries as intermediaries that allow parallel processing to proceed without blocking on missing child file data. These dummy entries act as placeholders that maintain data structure integrity and accuracy while enabling other parts of the system to process in parallel, improving overall productivity.
3Reliability
If remote storage devices are used for cross-region replication, then data security and availability are improved, but IO traffic increases and performance decreases
Solution Approach 1:
The patent segments metadata access patterns by separating parent directory B-trees from child file B-trees. This allows the system to cache frequently accessed parent directory information locally in memory, reducing the need for repeated remote IO operations and decreasing IO traffic while maintaining data availability through the hierarchical cache structure.
Solution Approach 2:
The patent performs preliminary action by pre-loading and caching parent directory B-tree data into memory before it is needed for child file operations. This preliminary caching reduces subsequent remote IO traffic to storage devices, improving performance while maintaining data security and availability through the hierarchical cache architecture.
Data Source
AI summary
Techniques are described for a hierarchical caching mechanism enabling efficient cross-region replications. In some embodiments, replication-related information (e.g., key-value pairs) is stored in a particular layout in a binary tree (B-tree) of a file system for replication processing. A hierarchy of caches storing a first type of information (e.g., crypto keys associated with iNodes) may be arranged to match the particular layout in the B-tree to enable efficient parallel processing of a second type of information (e.g., files, file data, or symbolic links), where the replication-related information in the B-tree is partitioned into multiple key ranges for parallel processing. In some embodiments, the caches in different hierarchies may be shared by different parallel-processing key ranges and replication jobs in a file system.