Watermark-Based Change Data Capture for Concurrent Log and Dump Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional multi-datastore synchronization techniques face limitations such as datastores remaining out of sync, propagation delays, disruption of real-time changes, and reliance on non-transferrable advanced database features, especially in heterogeneous environments.
Innovation Solution
Concurrent log and dump processing using watermark-based techniques that allow for the selection and transmission of non-conflicting data chunks, enabling real-time event propagation and full state capture across multiple databases without requiring locks or restarts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional multi-datastore synchronization techniques (dual-writes, distributed transactions) are used, then data consistency between datastores can be maintained, but the system complexity increases and propagation delays occur
Solution Approach 1:
The patent extracts the synchronization logic from complex distributed transaction protocols and implements a simpler event-driven architecture using CDC. Instead of coordinating multiple datastores through complex transactions, the system captures changes from the source datastore and propagates them independently to derived datastores, eliminating the need for complex coordination while maintaining consistency
Solution Approach 2:
The patent introduces an intermediary CDC mechanism that sits between the source and derived datastores. This intermediary captures transaction logs and propagates changes without requiring direct coordination between datastores, simplifying the synchronization architecture while maintaining data consistency across heterogeneous systems
2Reliability
If dump processing is executed during bootstrap phase or when data loss is detected, then data completeness can be ensured, but real-time event propagation is disrupted
Solution Approach 1:
The patent performs dump processing as a preliminary action during the bootstrap phase to establish initial data completeness before real-time CDC begins. Once the initial dump is complete, the system switches to continuous log-based CDC without stopping propagation, ensuring both data completeness and uninterrupted real-time event propagation
Solution Approach 2:
The patent ensures continuous CDC operation by performing dump processing in the background without halting log event processing. The system maintains continuous propagation of real-time changes while periodically executing dumps to ensure data completeness, eliminating disruptions to real-time event flow
3Reliability
If table locking is used during dump processing, then data consistency during dump can be ensured, but write traffic is blocked and real-time change propagation is disrupted
Solution Approach 1:
The patent replaces the mechanical table locking mechanism with a watermark-based versioning system. Instead of blocking writes with locks, the system uses watermarks to track the state of data at specific points in time, allowing concurrent reads and writes without blocking while maintaining dump consistency through version control
Solution Approach 2:
The patent introduces watermark tables as intermediaries that record the state of data at specific log positions. These watermark tables allow the dump process to read consistent snapshots without blocking write operations to the actual data tables, enabling concurrent dump processing and real-time change propagation
4Productivity
If advanced database features are used for log processing, then processing efficiency can be improved, but portability across different database systems is lost
Solution Approach 1:
The patent implements a universal CDC framework that works across multiple database systems by using standard SQL features and transaction log mechanisms that are commonly available in most databases. The system abstracts database-specific details behind a unified interface, maintaining processing efficiency through optimized log parsing while ensuring portability across heterogeneous database environments
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Various embodiments set forth systems and techniques for concurrent log and dump processing. The techniques include selecting, from a datastore, a chunk comprising one or more rows of data; comparing the one or more rows of data in the chunk with a first set of log events in a change log associated with the datastore, wherein each log event included in the first set of log events occurs after a first log event in the change log and prior to a second log event in the change log; selecting, based on the comparison, one or more non-conflicting rows in the chunk; and transmitting, to an output, one or more log events associated with the one or more non-conflicting rows in the chunk prior to processing a second set of log events in the change log, wherein the second set of log events occur after the second log event.