Multi-database log with DAG replication for consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing applications face challenges in maintaining high performance and low latency due to issues with state information management, such as locking mechanisms becoming bottlenecks and 'stop-the-world' pauses, especially in complex and large-scale environments.
Innovation Solution
A dynamic directed acyclic graph (DAG) of replication nodes is used to manage application state changes, allowing for fault-tolerant and efficient propagation of state transitions without requiring synchronization of all nodes, using acceptor, intermediary, and committer nodes, and a configuration manager that handles configuration changes asynchronously.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking mechanisms are used to manage application state changes, then consistency is improved, but performance becomes bottlenecked
Solution Approach 1:
The patent extracts the locking mechanism from the state management process entirely. Instead of using locks to ensure consistency, the system uses a log-based approach where state changes are recorded sequentially and applied by multiple database instances independently, eliminating the performance bottleneck while maintaining consistency through the ordered log replay mechanism
Solution Approach 2:
The patent introduces a log as an intermediary between the application state changes and the database instances. The log captures state transitions and distributes them to multiple databases, which then apply changes independently without needing to coordinate through locks, thus maintaining consistency without performance degradation
2Reliability
If locking mechanisms are used to manage application state changes, then consistency is improved, but latency increases
Solution Approach 1:
The patent removes locking mechanisms from the state management process, eliminating the latency associated with acquiring and releasing locks. Consistency is maintained through the sequential log structure and ordered replay mechanism rather than through time-consuming lock coordination
Solution Approach 2:
The patent performs preliminary actions by capturing state changes in the log before distributing them to multiple database instances. This allows databases to process changes independently at their own pace without waiting for lock coordination, reducing overall latency while maintaining consistency through the pre-established log order
3Reliability
If stop-the-world pauses are used to propagate state changes, then consistency is improved, but availability deteriorates
Solution Approach 1:
The patent performs preliminary action by recording all state changes in the log before propagation is needed. This allows multiple database instances to independently replay the log and achieve consistency without requiring coordinated stop-the-world pauses, maintaining continuous availability
Solution Approach 2:
The patent enables each database instance to independently process and apply log entries without requiring coordination or pauses with other instances. Each database serves itself by replaying the log locally, achieving consistency autonomously while maintaining continuous service availability
4Reliability
If synchronous replication is used across all nodes, then consistency is improved, but performance deteriorates
Solution Approach 1:
The patent segments the replication process into independent database instances that each process the log independently. Instead of requiring all nodes to synchronize together, each instance replays the log sequence locally, maintaining consistency through the ordered log structure while achieving parallel processing and higher throughput
Solution Approach 2:
The patent applies partial action by having each database instance process only the relevant portion of the log sequence independently. This allows parallel processing across multiple instances without requiring full synchronization, maintaining consistency through the log order while achieving excessive throughput through concurrent operations
Data Source
AI summary
At a logging service, a representation of a transaction requested by a client is received. The transaction includes at least a first write operation directed to a data object at a first data store. Based at least in part on the contents of the representation and contents of a persistent log of the logging service, a determination is made that the requested transaction is to be committed. A log record corresponding to the first write operation is inserted into the persistent log. The first write is propagated to the first data store after the record has been inserted in the persistent log.