Distributed Coordination Engine for Real-Time Data Consistency Checks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems struggle to ensure data consistency across distributed, heterogeneous storage environments, particularly in ensuring one-copy equivalence and consistency across geographically dispersed systems like AWS and Azure, which operate asynchronously and at varying speeds.
Innovation Solution
The Distributed Coordination Engine (DConE) implements an enhanced Paxos algorithm to achieve global transaction ordering and automated recovery, ensuring that data mutations are applied consistently across multiple storage systems by using a global sequence number (GSN) and active-active replication, allowing for real-time consistency checks and automated recovery from network outages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across distributed heterogeneous storage systems, then data availability and redundancy are improved, but data consistency and equivalence across systems deteriorate
Solution Approach 1:
The patent introduces a coordination engine as an intermediary component that mediates between distributed storage systems. This engine receives data change notifications from different storage systems, translates them into a common format, and coordinates the application of these changes across all systems to maintain consistency while allowing replication for availability.
Solution Approach 2:
The patent changes the parameter of data equivalence from a local property to a global property by implementing a coordination mechanism that tracks and synchronizes data change parameters across distributed systems. This allows the system to maintain data consistency through parameter synchronization while preserving replication benefits.
2Manufacturing precision
If local consistency checks are performed in each storage system, then local data integrity is improved, but global consistency detection across distributed systems deteriorates
Solution Approach 1:
The coordination engine performs multiple functions: it acts as a notification receiver from various storage systems, a translator of different data change formats, a coordinator of synchronized updates, and a detector of global consistency issues. This multi-functional approach enables both local integrity maintenance and global consistency detection through a single unified mechanism.
3Manufacturing precision
If real-time consistency checks are implemented across distributed systems, then data equivalence is improved, but system complexity and overhead increase
Solution Approach 1:
The system performs preliminary actions by establishing coordination mechanisms and notification subscriptions in advance. When data changes occur, the coordination engine has already been positioned to receive and process these changes, eliminating the need for complex real-time negotiation and reducing system overhead while maintaining data equivalence.
4Adaptability or versatility
If data replication occurs across heterogeneous storage systems, then system versatility is improved, but determining equivalence across different systems deteriorates
Solution Approach 1:
The coordination engine serves as an intermediary that translates data change notifications from heterogeneous storage systems into a universal format. This mediator capability allows the system to work with diverse storage technologies while maintaining the ability to determine equivalence through standardized change tracking and coordination.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer- implemented method may comprise executing, by a first plurality of replicated state machines, a sequence of ordered agreements to make mutations to a data stored in a first data storage service of a first type and executing, by a second plurality of replicated state machines, the sequence of ordered agreements to make mutations to the data stored in a second data storage service of a second type. First metadata of the mutated data stored in the first data storage service may then be received and stored, as may second metadata of the mutated data stored in the second data storage service. A synchronization may then be carried out as between the first and second data storage services using the received first and second metadata to determine when the data stored in the first data storage service that corresponds to the first metadata and the data stored in the second data storage service that corresponds to the second metadata have both settled after having mutated according to a predetermined one of the sequence of ordered agreements. A comparison of the stored first and second metadata may then be carried out when the data stored in the first data storage service that corresponds to the first metadata and the data stored in the second data storage service that corresponds to the second metadata have been determined to have settled according to the predetermined one of the sequence of ordered agreements. A selected action may then be carried out depending upon a result of the comparison.