Row-Level Change Stream Tracking for Multi-Master NoSQL Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for replicating changes in multi-region, multi-master distributed databases face challenges with handling interruptions and discrepancies between clusters, leading to issues like duplication or missing changes due to asynchronous replication methods relying on timestamps.
Innovation Solution
Implement a replication module that tracks change stream positions at a granular level (row or partition level) in the replication log, maintaining accurate replication status across clusters using change stream positions and incarnations to handle repartitioning and node changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If asynchronous replication is used to replicate changes between clusters, then replication speed and system availability are improved, but data consistency and reliability deteriorate due to interruptions and discrepancies
Solution Approach 1:
The patent implements feedback mechanisms through change stream positions that track the state of replication for each row. When interruptions occur or discrepancies are detected between clusters, the system uses these position markers to detect what changes have been replicated and what have not, enabling automatic recovery and consistency verification without manual intervention
Solution Approach 2:
The system performs preliminary actions by maintaining change stream positions that record the expected state of replication before interruptions occur. These pre-established position markers enable the system to quickly identify and recover from replication failures without needing to re-synchronize entire data sets, thus maintaining both speed and reliability
2Reliability
If change data capture state tracking is implemented at a granular level, then data consistency and reliability are improved, but device complexity and operational overhead increase
Solution Approach 1:
The patent applies segmentation by dividing the change tracking into row-level change stream positions rather than cluster-level or table-level tracking. Each row maintains its own position marker in the replication log, allowing the system to track replication status independently for each row. This granular segmentation improves replication accuracy while managing complexity through modular, independent tracking units
Solution Approach 2:
The system uses copying by maintaining change stream positions that are replicated alongside the actual data changes. These position markers are copied from the source cluster to the target cluster as part of the replication process, providing a lightweight mechanism for tracking state without requiring complex synchronization protocols or additional computational overhead
Data Source
AI summary
A method for change data capture state tracking includes accessing a distributed database including a plurality of clusters, each cluster of the plurality of clusters including a respective plurality of partitions, each cluster of the plurality of clusters configured to receive read and write operation. The method includes receiving, at a second cluster, a plurality of changes for a second table and storing the plurality of changes at a replication log. The method also includes asynchronously replicating the plurality of changes from the second table to a first table and maintaining a respective change stream position tracking a respective position in the replication log indicating one or more changes of the plurality of changes that have been replicated. The method includes receiving a read request at the first cluster requesting one or more rows of the first table and returning the respective change stream position.