Snapshot Key Inter-dependency Resolution in Cross-Region Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file system cross-region replication technologies face inefficiencies due to dependency relationships between parent directories and child files, leading to high IO traffic, long latency, and hindered parallel processing, especially when storage devices are remote, and lack effective mechanisms for snapshot key inter-dependency resolution.
Innovation Solution
A hierarchical caching mechanism is introduced, utilizing a B-tree layout for replication-related information and a hierarchy of caches to enable efficient parallel processing and inter-dependency resolution, where a dummy entry is created in the B-tree to allow processing of second-type information without waiting for the first-type information, and crypto keys are reused and updated across caches for encryption and decryption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional replication methods are used with dependency relationships between parent directories and child files, then data consistency is maintained, but IO traffic increases, latency increases, and parallel processing is hindered
Solution Approach 1:
The patent segments the B-tree structure into multiple cache levels (L1, L2, L3 caches) that can be processed independently. Each cache level stores specific types of information (e.g., L1 for parent directory keys, L2 for child file keys, L3 for data blocks), allowing parallel processing of different segments without waiting for dependencies to be resolved sequentially.
Solution Approach 2:
The patent performs preliminary actions by pre-loading and caching metadata information in hierarchical cache structures before actual replication operations. The B-tree keys and metadata are organized in advance across multiple cache levels, so that during replication, the system can directly access pre-positioned data without real-time dependency resolution, enabling parallel processing while maintaining consistency.
2Reliability
If storage devices are remote, then data security and availability are improved, but IO traffic increases and latency increases
Solution Approach 1:
The patent implements a nested hierarchical cache structure where L1 cache is nested within L2 cache, and L2 is nested within L3 cache. This nested arrangement allows the system to access frequently used metadata and keys in faster L1/L2 caches before falling back to remote storage, reducing the need for frequent remote IO operations and thereby reducing latency while maintaining data availability.
Solution Approach 2:
The patent introduces a temporal dimension to storage by creating a hierarchical time-based cache structure. Recently accessed and frequently used data are kept in faster, closer cache levels, while less frequently accessed data remain in remote storage. This dimensional organization allows the system to optimize access patterns based on temporal usage patterns, reducing latency for active operations.
3Productivity
If hierarchical caching mechanism with B-tree layout is used, then parallel processing efficiency and throughput are improved, but device complexity increases
Solution Approach 1:
The patent designs the hierarchical cache structure to serve multiple functions simultaneously: it acts as a B-tree organization structure, a caching mechanism, and a parallel processing framework. The same L1/L2/L3 cache hierarchy is used for both metadata storage and data block storage, eliminating the need for separate complex structures and reducing overall system complexity while enabling parallel processing.
Solution Approach 2:
The hierarchical cache structure is designed to be self-organizing and self-managing. The B-tree keys automatically route to appropriate cache levels based on their type and access patterns, without requiring complex external management. The system automatically manages cache coherence and data placement across levels, reducing the operational complexity despite the multi-level architecture.
Data Source
AI summary
Techniques are described for snapshot key inter-dependency resolution during cross-region replications. Dependency between a first type of replication-related information (e.g., crypto keys associated with a parent directory iNode or a file iNode) and a second type of replication-related information (e.g., files, file data/FMAPs, or symbolic links) during a cross-region replication may be resolved to enable non-blocking delta application in a target file system. In some embodiments, temporary dummy entries for the first type of information may be created in the B-tree of the target file system for the out-of-order download (e.g., the second type being downloaded before the first type) of these two types of information. In some embodiments, a consolidation process may be performed between the dummy entries and the later-arriving first type of information.