Data Replication System Using Change Metadata and Shadow Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data replication from one system to another is a slow and resource-intensive process, especially with large data sets, as it involves not only initial data copying but also tracking and transferring changes, which consumes additional time and resources.
Innovation Solution
The Data Replication System (DRS) employs triggers to log minimal delta changes during the initial load, assembling and transferring only the most recent updates to a shadow database, which are then merged with the target database upon completion, optimizing resource usage and reducing replication time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication is performed by copying all data and tracking all changes, then data consistency is maintained, but replication time and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the essential change information (commit hashes, parent hashes, base hashes) from the original data, storing minimal metadata in the change table rather than replicating entire data objects. This selective extraction maintains data consistency through change tracking while dramatically reducing replication time and resource consumption.
Solution Approach 2:
Instead of copying complete data objects and tracking changes on the full copies, the patent inverts the approach by storing only change metadata and reconstructing data objects on-demand at the target system. This inversion transforms the replication process from heavy data movement to lightweight metadata synchronization.
2Manufacturing precision
If all data changes are tracked and transferred during initial load, then data accuracy is ensured, but resource consumption increases
Solution Approach 1:
The patent extracts only the essential change information (commit hashes, parent hashes, base hashes) from the original data, storing minimal metadata in the change table rather than replicating entire data objects. This selective extraction maintains data consistency through change tracking while dramatically reducing replication time and resource consumption.
Solution Approach 2:
The patent creates lightweight copies of change metadata (commit hashes, parent hashes, base hashes) rather than copying complete data objects. These metadata copies are stored in the change table and used to reconstruct data objects on-demand, reducing resource consumption while ensuring data accuracy through verifiable change tracking.
3Reliability
If change tracking is implemented during data copying, then data consistency is maintained, but the process becomes more complex
Solution Approach 1:
The patent introduces a change table as an intermediary component that stores minimal change metadata (commit hashes, parent hashes, base hashes). This intermediary simplifies the replication process by decoupling change tracking from data copying, maintaining data consistency through a manageable metadata layer rather than complex synchronization logic.
Solution Approach 2:
The patent changes the parameters of change tracking from storing complete data objects to storing only essential metadata (commit hashes, parent hashes, base hashes). This parameter transformation reduces system complexity by working with compact, standardized data structures instead of complex data objects during the replication process.
Data Source
AI summary
Disclosed herein are various embodiments for a data replication system. An embodiment operates by determining that a plurality of records of a source table are copied from a source system to a target table of a target system. It is determined that a change table, of the source system, associated with the source table is populated with a plurality of changes occurring to at least a subset of records of the plurality of records while the plurality of records are being copied from the source table of the source system to the target table of the target system. It is determined that the plurality of changes are copied to a shadow table of the target system. It is determined that the target table includes the plurality of records from the source table. The plurality of changes of the shadow table are merged with the target table.