Log-Coordinated Storage Group Configuration via DAG Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed applications face challenges in maintaining high performance and consistency as they scale, due to issues like locking mechanisms becoming bottlenecks and 'stop-the-world' pauses, especially in latency-sensitive environments with complex network configurations and high state transition rates.
Innovation Solution
A directed acyclic graph (DAG) of replication nodes is used to manage application state changes, allowing for fault-tolerant and efficient propagation of state transitions without requiring synchronization, using acceptor, intermediary, and committer nodes, and a configuration manager that propagates configuration-delta messages asynchronously to maintain consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking mechanisms are used to manage application state changes, then consistency is improved, but performance becomes bottlenecked
Solution Approach 1:
The patent extracts the locking mechanism from the state management process entirely. Instead of using locks to ensure consistency, the system uses a log-based approach where state changes are recorded sequentially and applied in order, eliminating the need for locking while maintaining consistency across distributed nodes.
Solution Approach 2:
The patent replaces the mechanical locking system with a log-based eventual consistency model. State changes are propagated through logging and sequential application rather than through locking mechanisms, substituting a different architectural approach that avoids the performance bottlenecks of traditional locking.
2Reliability
If state information is propagated among application components, then consistency is improved, but normal operations must be paused
Solution Approach 1:
The patent performs preliminary actions by logging state changes before they are fully propagated to all nodes. The log records the state transition in advance, allowing nodes to apply changes asynchronously without pausing operations. This preliminary logging enables eventual consistency without stop-the-world pauses.
Solution Approach 2:
The patent maintains continuity of useful action by allowing the application to continue processing requests while state changes are propagated in the background through the logging mechanism. Normal operations continue uninterrupted while consistency is gradually achieved across the distributed system.
3Reliability
If configuration changes are propagated synchronously, then consistency is improved, but handling high rates of state transitions becomes bottlenecked
Solution Approach 1:
The patent implements periodic action by batching configuration changes and propagating them at intervals rather than synchronously with each state transition. The logging mechanism accumulates state changes and propagates them periodically, allowing high rates of state transitions to be handled without blocking while maintaining consistency through the ordered log application.
Data Source
Figure 1
Figure 2a~2c
Figure 2d~2f
AI summary
A configuration manager of a storage service receives a set of service requirements, comprising one or more of: a performance requirement for one or more types of storage operations, or an access interface type requirement Based on the service requirements, a candidate storage configuration that includes one or more data store instances and a first log-based transaction manager is generated. Subsequent to an approval of the first storage configuration by a client, the establishment of the data store instances and the log-based transaction manager is initiated.