Log-Coordinated Storage Cross-Data Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing environments, managing application state changes becomes challenging due to increased complexity and size, leading to performance bottlenecks and latency issues, especially in mission-critical workloads across multiple time zones, as traditional techniques either lock state information or require 'stop-the-world' pauses to propagate changes.
Innovation Solution
A dynamic directed acyclic graph (DAG) of replication nodes is used to manage application state changes, allowing asynchronous propagation of configuration-delta messages and avoiding the need for synchronization, enabling continuous processing even with divergent views of the DAG configuration, and implementing a coordinated suspension technique for efficient recovery from failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional locking mechanisms are used to manage application state changes, then consistency is maintained, but performance becomes a bottleneck as application size and complexity increase
Solution Approach 1:
The patent extracts the locking mechanism from the state management process entirely. Instead of using locks to ensure consistency, the system uses a logging service that records state changes in a log and propagates them asynchronously to multiple data stores. This removes the performance bottleneck caused by locking while maintaining consistency through the log-based approach.
Solution Approach 2:
The logging service acts as an intermediary between the application and the data stores. It receives state changes from the application, records them in a log, and propagates them to multiple data stores asynchronously. This intermediary approach eliminates the need for direct locking between data stores while ensuring consistency through the centralized log.
2Productivity
If locking mechanisms are avoided to improve performance, then throughput increases, but 'stop-the-world' pauses are required to propagate state changes
Solution Approach 1:
The patent implements continuous propagation of state changes through asynchronous processing. The logging service continuously reads from the log and propagates state changes to data stores without requiring the application to stop. This eliminates 'stop-the-world' pauses while maintaining high throughput, as the propagation occurs in the background independently of application execution.
3Reliability
If asynchronous propagation is used to eliminate stop-the-world pauses, then availability improves, but handling very high rates of state transitions becomes bottlenecked
Solution Approach 1:
The patent segments the state transition handling into multiple independent components: the logging service that receives and records transitions, and multiple data stores that process them asynchronously. This segmentation allows the system to handle very high rates of state transitions by distributing the processing load across multiple independent data stores that can operate in parallel without interfering with each other.
4Reliability
If synchronized propagation is used to ensure consistency, then data integrity is maintained, but downtime increases during configuration changes
Solution Approach 1:
The patent uses preliminary action by pre-recording all state changes in the log before propagation to data stores. The logging service maintains a persistent log of state transitions, ensuring that even if propagation is interrupted or data stores are restarted, the changes can be recovered and applied. This preliminary recording ensures data integrity without requiring synchronized propagation, thereby eliminating downtime during configuration changes.
Data Source
AI summary
Data stores that are to be configured as members of a storage group on behalf of a client are identified. The client indicates one or more types of cross-data-store operations to be performed. A transaction manager that makes acceptance decisions for write requests directed to the members based at least partly on contents of a log is instantiated, and respective write transformers are set up for at least some of the types of cross-data-store operations. During a particular time period, the workload at the storage group is monitored, and the number and types of cross-data-store operations is determined. The results of the monitoring are used to determine a billing amount for the client based on a selected pricing policy.