Multi-database Log Transaction Replication via DAG
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, managing application state changes while maintaining high performance and low latency is challenging due to issues with locking mechanisms and 'stop-the-world' pauses, especially in large and complex systems, where failure events and network delays are common.
Innovation Solution
A dynamic directed acyclic graph (DAG) of replication nodes is used to manage application state changes, allowing for fault-tolerant and asynchronous propagation of state transitions without requiring synchronization of all nodes, thereby avoiding 'stop-the-world' scenarios and minimizing downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking mechanisms are used to manage application state changes, then consistency is improved, but performance becomes a bottleneck
Solution Approach 1:
The patent extracts the locking mechanism from the state management process entirely. Instead of using locks to ensure consistency, the system uses version vectors and conflict detection to manage state changes asynchronously, allowing multiple nodes to operate independently without blocking each other.
Solution Approach 2:
The patent introduces version vectors as an intermediary mechanism to track state changes across distributed nodes. These version vectors enable the system to detect conflicts and resolve them without requiring direct locking between nodes, thus maintaining consistency while avoiding performance bottlenecks.
2Reliability
If locking mechanisms are used to manage application state changes, then consistency is improved, but latency increases
Solution Approach 1:
The patent removes locking mechanisms from the state management process, eliminating the wait times and blocking operations that cause latency. Nodes can process state changes independently and immediately, without being blocked by locks held by other nodes.
Solution Approach 2:
The patent enables continuous operation of distributed nodes by allowing them to process state changes asynchronously without interruption. The conflict detection and resolution mechanisms operate in the background, allowing useful work to continue uninterrupted while maintaining consistency.
3Reliability
If stop-the-world pauses are used to propagate state changes, then consistency is improved, but availability deteriorates
Solution Approach 1:
The patent extracts the state propagation process from the operational workflow. Instead of pausing operations to propagate state changes, the system uses asynchronous replication with conflict detection, allowing nodes to continue operating while state changes are propagated in the background.
Solution Approach 2:
The patent performs preliminary conflict detection and resolution preparation before state changes need to be applied. Version vectors are maintained continuously, allowing the system to quickly detect and resolve conflicts without requiring stop-the-world pauses when state propagation is needed.
4Reliability
If synchronous node synchronization is used, then consistency is improved, but performance deteriorates
Solution Approach 1:
The patent segments the state management process into independent node operations with asynchronous communication. Each node independently processes state changes and maintains its own version vector, eliminating the need for synchronous coordination and allowing parallel processing across multiple nodes.
Solution Approach 2:
The patent enables continuous state change processing across all nodes simultaneously. The asynchronous conflict detection and resolution mechanisms allow nodes to operate independently and continuously, maximizing throughput while maintaining consistency through version vector tracking.
Data Source
Figure 1
Figure 2a~2b
Figure 2d~2f
AI summary
At a logging service, a representation of a transaction requested by a client is received. The transaction includes at least a first write operation directed to a data object at a first data store. Based at least in part on the contents of the representation and contents of a persistent log of the logging service, a determination is made that the requested transaction is to be committed. A log record corresponding to the first write operation is inserted into the persistent log. The first write is propagated to the first data store after the record has been inserted in the persistent log.