Tracker Table Intent Records for Consistent Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage environments, ensuring data consistency across replicated data stores is challenging due to increased accessibility, which often compromises scalability and is technology-agnostic, making existing methods inefficient.
Innovation Solution
A technology-agnostic system employing a tracker table with intent records containing primary keys and versions to manage write operations, reconcile non-propagated changes, and handle deletions, ensuring eventual consistency across primary and secondary data stores without relying on specific storage technologies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If multiple replicas of data are included in a distributed storage system to improve data accessibility, then data accessibility is significantly improved, but data consistency is compromised
Solution Approach 1:
The system performs preliminary actions by capturing write intents before actual data propagation occurs. Intent records are created and stored in a tracker table, documenting the desired state changes. This allows the system to plan and coordinate data replication across multiple replicas in advance, ensuring consistency is maintained while enabling parallel propagation to improve accessibility.
Solution Approach 2:
The patent introduces an intermediary reconciliation mechanism that mediates between primary and secondary data stores. The reconciliation process acts as a mediator that detects and resolves consistency issues arising from failed propagations, allowing multiple replicas to be accessed simultaneously while maintaining data integrity through coordinated correction of inconsistent states.
2Reliability
If traditional data replication methods are used to ensure data consistency, then data consistency is maintained, but scalability is compromised
Solution Approach 1:
The system segments the data replication process into independent, manageable components: intent capture, data propagation, and reconciliation. Each component operates semi-independently, allowing the system to scale by adding more replicas without requiring complex coordination across the entire system. The tracker table segments the tracking of multiple data changes into discrete intent records that can be processed individually.
Solution Approach 2:
The reconciliation process operates in a self-service manner, automatically detecting and correcting consistency issues without requiring manual intervention or complex external coordination. The system monitors its own state and performs self-correction through the reconciliation mechanism, enabling scalable operation as the number of replicas increases without proportionally increasing management overhead.
3Reliability
If technology-specific replication methods are employed to maintain consistency, then data consistency is ensured, but the system becomes inefficient and less adaptable to diverse storage technologies
Solution Approach 1:
The patent implements a universal replication framework that can operate with diverse storage technologies through a common interface. The tracker table and intent record structure provide a technology-agnostic mechanism for capturing and propagating data changes, while the reconciliation process works uniformly across different storage backends. This multi-functional approach allows the same system to support various storage technologies without sacrificing consistency or efficiency.
Data Source
AI summary
A method for achieving eventually-consistent data replication in a technology agnostic manner. The method can be implemented via execution of computing instructions configured to run at one or more processors and stored at one or more non-transitory computer-readable media. The method can include receiving, by a system comprising the one or more processors and the one or more non-transitory computer-readable media, a write request for an object. The write request can be received from a user device or an application server. The method further can include performing a write operation for the object, by the system, after the write request is received. Performing the write operation can comprise: (a) adding an intent record to a tracker table, the intent record comprising a primary key uniquely associated with the object and a version associated with the object; and (b) updating the object, the object including the primary key and the version, in a primary data store, after the intent record is added. The method further can include reconciling, by the system, each intent record that is stranded in the tracker table. Reconciling each intent record that is stranded in the tracker table can comprise, for a respective stranded intent record of the one or more stranded intent records, (a) attempting to obtain, from the primary data store, a primary object that is uniquely associated with a stranded primary key of the respective stranded intent record; (b) when (A) the primary object is obtained and (B) a primary object version of the primary object is equal to a stranded intent version of the respective stranded intent record, replicating the primary object from the primary data store to at least one secondary data store; (c) when the primary object is not obtained, deleting the object from the secondary data store; and (d) when at least one of: (A) the primary object is not obtained, (B) the primary object version of the primary object is not equal to the stranded intent version of the respective stranded intent record, or (C) the primary object from the primary data store is replicated to the at least one secondary data store, clearing the respective stranded intent record from the tracker table. Other embodiments are disclosed.