Database Replication Conflict Resolution via Dual-Stream Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High incoming rates of incremental changes in database replication can lead to significant delays in achieving a steady state between source and replica sites, as the instantiation process takes a long time, resulting in stale data and requiring suspension of incremental data change transfers and new table instantiation.
Innovation Solution
Implementing two streams of data operations - an instantiation stream and an apply stream - that run concurrently, with automatic conflict resolution mechanisms to continuously apply incremental changes while bulk instantiation occurs, allowing new tables to be added without suspending existing streams and ensuring the replica site reaches a steady state.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bulk instantiation is performed to replicate data from source to target database, then data replication is achieved, but the replication process takes a long time causing data staleness
Solution Approach 1:
The patent segments the data replication process into two independent streams: instantiation stream (bulk data loading) and apply stream (incremental changes). These streams operate concurrently and independently, allowing the system to replicate both historical data and incremental changes without waiting for one process to complete before starting the other, thereby reducing total replication time while maintaining data consistency.
Solution Approach 2:
The patent performs preliminary actions by pre-instantiating the database structure and historical data through the instantiation stream before incremental changes are applied. This allows the target database to be ready to receive incremental changes earlier, reducing the overall time to achieve current data state while ensuring consistency through conflict resolution mechanisms.
2Reliability
If incremental changes are captured and transferred continuously, then data freshness is improved, but conflicts arise between instantiation and apply streams requiring suspension
Solution Approach 1:
The patent implements feedback mechanisms through conflict detection and resolution logic that monitors both streams for conflicting operations. When conflicts are detected (e.g., simultaneous insert/delete/update of same row), the system automatically resolves them by determining operation priority based on timestamps or operation types, allowing continuous replication without suspension while maintaining data consistency.
Solution Approach 2:
The patent introduces an intermediary conflict resolution layer that mediates between the instantiation stream and apply stream. This intermediary component analyzes conflicting operations, determines which operation should take precedence, and applies appropriate resolution strategies (such as ignoring lower-priority operations or merging conflicts), enabling continuous data replication without manual intervention or stream suspension.
3Reliability
If new tables are instantiated in the first stage, then data structure replication is complete, but the second stage must be suspended for table creation delays
Solution Approach 1:
The patent segments schema replication into two parts: structure instantiation through the instantiation stream and incremental DDL operations through the apply stream. This allows new tables and schemas to be created continuously in the target database as they appear in the source, without suspending the incremental data change transfer process, thereby maintaining both schema consistency and high replication throughput.
Solution Approach 2:
The patent ensures continuity of useful action by allowing the apply stream to continue capturing and transferring incremental changes (including new table creations) without suspension. The system processes DDL operations concurrently with DML operations through the dual-stream architecture, eliminating idle time and maintaining continuous data synchronization between source and target databases.
Data Source
AI summary
An approach is provided to automatically resolve conflict between instantiation and ongoing data replication operations. In an embodiment, incremental change records that describe one or more database operations are received at a sink entity. Here, all of the one or more database operations have occurred in a second database after a particular time. A first database operation in the one or more database operations to a particular row in the first database is applied. When a conflict with a second database operation that operates on the particular row is detected, one or more compensating actions, which may, but are not limited to, be a different database operation and/or an operation that creates a conflict log entry, may be performed. Here, the second database operation has occurred in the second database prior to the particular time. When the second database operation is to be performed, an appropriate operation may be performed.