CDC Log Coherence via Snapshot Delta Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Change Data Capture (CDC) systems face challenges in maintaining coherence between source and target databases, especially when snapshots are updated, leading to incoherent data replication across systems, particularly in hybrid cloud environments where wide area networks and high-latency connections complicate data synchronization.
Innovation Solution
A method is introduced to generate coherent CDC logs by comparing initial and subsequent snapshots to derive corrective operations, ensuring that the target system reflects the state of the source system at a specific snapshot time, while allowing only changed data to be processed, thus maintaining coherence without requiring a full re-read of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full snapshots are used to ensure data coherence in CDC systems, then data consistency between source and target systems is improved, but the amount of data to be processed increases significantly
Solution Approach 1:
The patent extracts only the changed data portions from full snapshots by comparing consecutive snapshots and identifying differences. Instead of processing complete snapshots, the system extracts and processes only the delta changes, maintaining coherence while reducing data volume significantly.
Solution Approach 2:
The patent segments the data synchronization process into multiple snapshots taken at different time points. By dividing the continuous data stream into discrete snapshot segments and comparing them, the system identifies only the changes between segments, avoiding processing of unchanged data.
2Productivity
If consecutive snapshots are compared to identify changes, then processing efficiency is improved by avoiding full data re-processing, but the complexity of tracking and comparing snapshots increases
Solution Approach 1:
The patent implements a feedback mechanism where each snapshot comparison result feeds into the next comparison operation. The system maintains a history of snapshots and uses previous comparison results to inform subsequent comparisons, creating an efficient feedback loop that reduces redundant processing while managing complexity through structured state tracking.
3Reliability
If CDC systems process all data changes including initial refresh, then complete data replication is achieved, but the time required for synchronization increases
Solution Approach 1:
The patent performs preliminary actions by taking an initial snapshot before the CDC process begins and storing it for later comparison. This preliminary snapshot serves as a baseline that eliminates the need to re-process initial data during subsequent synchronization operations, reducing overall synchronization time while maintaining data completeness.
Data Source
AI summary
A method includes obtaining a first snapshot of a source system and deriving a set of key-value pairs reflecting the first snapshot. A mirror operation of the source system is performed to obtain CDC change operations. The method obtains a first CDC log as a first sequence of key-value pairs. A second snapshot of the source system is obtained and a set of key-value pairs is derived, which reflects the second snapshot. The first sequence of key-value pairs is compared with the set of key-value pairs to derive corrective CDC operations, captured as a set of key-value pairs. The corrective CDC operations represent corrections to be performed with respect to the first sequence of key-value pairs. A second CDC log is obtained as a second sequence of key-value pairs. The corrective CDC operations ensure that the second sequence of key-value pairs are coherent with the set of key-value pairs.