Data Synchronization via Partitioned Message Digests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current change data capture (CDC) methods require significant resources and data traffic between source and target systems, and often necessitate schema modifications in the source database, which can be inefficient and costly.
Innovation Solution
A method for synchronizing data between a source and target data set involves partitioning the data, calculating message digests for each partition, and comparing these digests to identify and capture changes, allowing for efficient data updates in the target data set without requiring extensive data traffic or schema modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire volume of data is migrated from source to target whenever source is updated, then data synchronization is achieved, but resource overhead becomes enormous
Solution Approach 1:
The patent divides the data set into multiple partitions and calculates message digests for each partition separately. This segmentation allows the system to identify and transfer only the specific partitions that have changed, rather than migrating the entire data volume, thereby reducing resource overhead while maintaining data synchronization.
Solution Approach 2:
The patent extracts only the changed data partitions from the source system based on message digest comparison. By taking out only the necessary changed portions rather than the entire data set, the system achieves data synchronization with minimal resource consumption and data traffic.
2Reliability
If CDC is implemented in source system using triggers and log files, then change data capture is achieved, but schema modifications are required
Solution Approach 1:
The patent creates a copy of the source data set in the target system and calculates message digests for comparison. This copying approach enables change detection without requiring triggers, log files, or schema modifications in the source system, as the message digest can be computed directly from the copied data.
Solution Approach 2:
The patent replaces the mechanical CDC mechanisms (triggers, log files, schema changes) with a computational approach using message digests. Instead of using complex source-side CDC infrastructure, the system uses cryptographic hash functions to detect changes, eliminating the need for source system modifications.
3Reliability
If message digest is calculated for entire data set, then change detection is achieved, but data traffic between source and target increases
Solution Approach 1:
The patent segments the data set into multiple partitions and calculates message digests for each partition separately. This allows the system to identify only the specific partitions that have changed, thereby reducing the quantity of data that needs to be transmitted between source and target systems while maintaining reliable change detection.
Solution Approach 2:
The patent applies partial action by calculating and comparing message digests only for the necessary partitions rather than the entire data set. This partial approach reduces data traffic to only the extent needed for effective change detection, avoiding the excessive data transmission that would result from processing the complete data volume.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces resource overhead and data traffic while ensuring accurate synchronization of data between the source and target systems, enhancing efficiency and reducing the need for schema modifications in the source database.
Implementation Method 1
a target message digest is calculated for the data present in each partition of the target data set. Based upon the partition definition of the target data set, the data present in the source data set is partitioned. A source message digest is calculated for the data present in each partition of the source data set.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are systems and methods for synchronizing data present in a target data set with data present in a source data set. The data present in the target data set is partitioned to determine a partition definition of the target data set. A target message digest is calculated for the data present in each partition of the target data set. Based upon the partition definition of the target data set, the data present in the source data set is partitioned. A source message digest is calculated for the data present in each partition of the source data set. The source message digest of a partition is compared with the target message digest of a corresponding partition to capture a change in data present in the source data set. Based upon the captured data, the data in the target data set is synchronized with the data in the source data set.