Asynchronous Data Replication With Identifier Annotation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage environments face challenges in geo-replicating data across geographically separate locations while ensuring data integrity and availability, particularly in scenarios where customers want to store data within specific geographical boundaries to avoid political or natural disaster risks.
Innovation Solution
The method involves annotating data with identifiers at a primary data store, committing it, and then replicating it to a secondary data store geographically distinct from the primary, where it is logged for eventual replay and commitment, with acknowledgments communicated between the stores to ensure data integrity and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous replication is used to ensure data is written to both primary and secondary stores before acknowledgment, then data reliability is improved, but response time deteriorates
Solution Approach 1:
The system performs preliminary action by writing data to the secondary store's log before receiving full acknowledgment, allowing the primary store to annotate data with identifiers and send acknowledgment earlier, while the secondary store completes replication asynchronously in the background
Solution Approach 2:
The system dynamically adjusts the replication process by allowing the secondary store to replay and commit data at its own pace after initial logging, creating a flexible asynchronous model where the primary store can acknowledge sooner while the secondary store catches up in the background
2Loss of time
If asynchronous replication is used to reduce latency by allowing early acknowledgment, then response time is improved, but data reliability deteriorates
Solution Approach 1:
The system implements feedback mechanisms where the primary store receives acknowledgment from the secondary store that data has been written to the log, allowing the primary store to confirm commitment to the client while the secondary store completes the full replication process in the background
Solution Approach 2:
The secondary store performs preliminary logging of data with identifiers before full commitment, allowing the primary store to send early acknowledgment while the secondary store completes replication asynchronously, maintaining both speed and reliability
3Reliability
If data is replicated to multiple geographical locations to reduce availability issues, then system reliability is improved, but device complexity deteriorates
Solution Approach 1:
The system segments the replication process into distinct phases: initial logging at the secondary store, identifier annotation at the primary store, and asynchronous replay/commitment at the secondary store, making the complex multi-location replication manageable through structured steps
Solution Approach 2:
The system uses identifiers as intermediaries that travel with data between primary and secondary stores, simplifying the coordination complexity by providing a clear reference mechanism for tracking and matching data across geographical locations
Data Source
Figure 1
Figure 2~5
Figure 3
AI summary
Embodiments of the present invention relate to asynchronously replicating data in a distributed computing environment. To achieve asynchronous replication, data received at a primary data store may be annotated with information, such as an identifier of the data. The annotated data may then be communicated to a secondary data store, which may then write the data and annotated information to one or more logs for eventual replay and committal at the secondary data store. The primary data store may communicate an acknowledgment of success in committing the data at the primary data store as well as of success in writing the data to the secondary data store. Additional embodiments may include committing the data at the secondary data store in response to receiving an instruction that authorizes committal of data through a identifier.