Change Data Capture for Database Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data synchronization methods between source and target database systems, especially in systems like OLTP and OLAP, face challenges in efficiently managing data changes due to differences in processing resources and data formats, leading to increased response times and resource utilization issues.
Innovation Solution
A change data capture method that configures the target database system to detect and apply data changes from the source dataset by processing log information, using asynchronous transaction log shipping, thread pooling, and batch processing to minimize resource usage and optimize data synchronization, even across different data formats like row-oriented and column-oriented storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full refresh systems copy entire data sets, then data consistency between source and target systems is maintained, but data transfer time and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the changed data portions from the source system using change data capture technology, rather than copying entire data sets. This is achieved by identifying and capturing specific data changes (inserts, updates, deletes) and applying only these changes to the target system, thereby maintaining data consistency while dramatically reducing transfer time and resource consumption.
2Productivity
If CDC systems apply only changed data, then data transfer efficiency improves, but complexity of detecting and applying changes increases
Solution Approach 1:
The patent introduces an intermediary change data capture service that sits between the source and target systems. This service automatically detects data changes in the source system, captures the change information, and applies it to the target system. The intermediary handles the complexity of change detection and application logic, simplifying the overall system architecture while maintaining high transfer efficiency.
3Use of energy by moving object
If target database system processes all synchronization operations, then source database system resource usage decreases, but target system processing time increases
Solution Approach 1:
The patent implements dynamic processing where the target database system is configured to handle change detection and application operations dynamically. The system adapts its processing capacity to handle synchronization operations, utilizing available resources efficiently. This dynamic approach allows the target system to process changes as they occur rather than batching them, reducing processing time while maintaining low source system resource usage.
Data Source
AI summary
According to one embodiment, a method, computer system, and computer program product for data synchronization between a source database system and a target database system, the source database system including a source dataset and the target database system including a target dataset that is configured to include a copy of the source dataset. The embodiment may include configuring the target database system to detect data changes in the source dataset. The embodiment may include applying the detected data changes on the target dataset.