Replication Message Queue Delayed Visibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, ensuring data replication across multiple servers for high availability and consistency while preventing stale data and handling transient errors and process crashes is challenging, as existing methods either lock all copies, leading to unavailability or result in indeterminate and lengthy operation loops.
Innovation Solution
A two-message approach is implemented, where a first replication message with delayed visibility ensures data persistence in a source storage, and a second message with immediate visibility replicates the data to target storages, optimizing replication latency and ensuring eventual consistency across all copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data replication is performed immediately after operation, then replication speed is improved, but data consistency and reliability deteriorate due to potential crashes or transient errors
Solution Approach 1:
The system performs preliminary persistence of the operation result in the source storage before creating the replication message. This ensures that the data is safely written and committed before being transmitted to target storages, preventing data loss in case of crashes during replication.
Solution Approach 2:
The patent extracts the persistence verification step from the replication process. By separating the persistence action (in source storage) from the replication action (to target storages), the system ensures reliability of the source data before initiating replication, thus resolving the contradiction between speed and reliability.
2Loss of time
If replication messages are sent with immediate visibility, then replication latency is reduced, but data consistency deteriorates due to potential stale data
Solution Approach 1:
The system performs the persistence action as a preliminary step before creating the replication message with immediate visibility. This ensures that the data is fully committed to the source storage before being made visible for replication, preventing stale data issues while maintaining low latency.
Solution Approach 2:
The system uses feedback from the persistence verification (checking whether the operation result was successfully persisted) to determine when to create the replication message. This feedback mechanism ensures data consistency before replication, resolving the contradiction between latency and consistency.
3Reliability
If all copies are locked during replication, then data consistency is improved, but availability deteriorates
Solution Approach 1:
The patent segments the replication process into distinct phases: persistence phase (in source storage) and replication phase (to target storages). By segmenting these operations, the system maintains consistency during persistence while allowing concurrent reads and writes to target storages, thus improving availability without sacrificing consistency.
Data Source
AI summary
Aspects of the present invention relate to guaranteeing delivery of replication messages in distributed storage systems. A request to perform an operation may be received. A first replication message corresponding to the request may be created, where the first replication message comprises instructions to replicate the result of the operation to one or more target storages. The replication message may be inserted in a message queue with a delayed visibility. The operation may be performed, where a result of the operation is persisted in a source storage in a data store. Upon determining that the result was successfully persisted in the source storage, a second replication message may be created, where the second replication message comprises instructions to replicate the result of the operation to the one or more target storages. The second replication message may be inserted in the message queue with an immediate visibility.