Asynchronous Metadata Replication for Distributed Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current disaster recovery and snapshot services face challenges in scaling with growing networks and services, as some persistent databases struggle to handle multiple data nodes, leading to data replication stalls and metadata loss due to reliance on external management systems and synchronous replication methods.
Innovation Solution
Implementing asynchronous local migration and remote replication of metadata between persistent data stores, allowing for independent operation of disaster recovery services at any site without external management systems, using a migration module to pause critical operations, replicate metadata, and store it in a new data store, enabling backwards compatibility and automatic conversion during migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous replication schemes are used with external management systems, then data consistency is maintained, but replication stalls occur and metadata loss happens when connections are dropped
Solution Approach 1:
The patent inverts the traditional synchronous replication approach by implementing asynchronous replication, where the primary site writes data locally first and then replicates to secondary sites without waiting for confirmation. This eliminates replication stalls caused by connection drops while maintaining data consistency through eventual consistency mechanisms and conflict resolution protocols.
Solution Approach 2:
The patent applies preliminary action by pre-synchronizing metadata inventories between sites before disaster recovery operations begin. This preliminary synchronization ensures that when replication resumes after connection restoration, data consistency is maintained without requiring lengthy re-synchronization processes, thus preventing metadata loss and replication stalls.
2Ease of operation
If external management systems are used for disaster recovery services, then centralized control is achieved, but the system cannot scale with growing networks and services
Solution Approach 1:
The patent segments the centralized management architecture into distributed autonomous nodes that can independently manage their own disaster recovery operations. Each site maintains local metadata inventories and can perform recovery operations autonomously, allowing the system to scale horizontally across growing networks without requiring a proportionally larger external management system.
Solution Approach 2:
The patent implements self-service mechanisms where each disaster recovery site automatically synchronizes its metadata inventory with other sites and can independently perform recovery operations. This eliminates the bottleneck of external management systems while maintaining coordination through automated peer-to-peer communication and conflict resolution protocols.
3Reliability
If persistent databases are used to store metadata for multiple data nodes, then data persistence is achieved, but the databases cannot handle growing numbers of nodes and services
Solution Approach 1:
The patent transitions from a single-dimension centralized database architecture to a multi-dimensional distributed metadata inventory system where each site maintains its own local metadata store. This dimensional change allows the system to handle growing numbers of nodes by distributing the metadata storage burden across multiple independent databases rather than concentrating it in a single database that becomes a bottleneck.
4Measurement precision
If synchronous metadata replication is implemented, then real-time consistency is maintained, but the system cannot support asynchronous operations or new persistence layers
Solution Approach 1:
The patent implements dynamic replication modes that can adapt between synchronous and asynchronous operations based on operational requirements. The system can switch between maintaining strict real-time consistency for critical metadata and allowing eventual consistency for less time-sensitive data, enabling support for both synchronous and asynchronous operations while maintaining adaptability to new persistence layers and operational patterns.
Data Source
AI summary
In one embodiment, systems and methods for performing asynchronous local migration of metadata between data stores and asynchronous remote replication of metadata between sites are described. The methods may use various configurations, including 1-to-1, 1-to-N, N-to-1, M-to-N, etc. The method for performing asynchronous local migration at a first site may include pausing critical operation(s) at an old data store, copying metadata from the old data store to a new data store, flagging table(s) in the old data store as complete, and deleting the metadata from the old data store. The method for asynchronous remote replication may include determining that local migration is complete, identifying second metadata from the new data store for which the first site is a primary authority, sending, to the second site, the second metadata, receiving, from the second site, third metadata for which the second site is the primary authority and storing the third metadata.