Source-Side Transaction Merging for Replication Order Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication engines face challenges in maintaining natural-flow order of transactions across multiple disks and systems, leading to inefficiencies and potential data integrity issues due to high transaction rates and multithreading, which can result in transaction events being received out of order at the target system.
Innovation Solution
Implementing a mechanism to resynchronize transaction events into natural-flow or landed order using a Router that sequences events based on timestamps or event positions, and pre-fetching change blocks from transaction-log disks to store in memory, allowing the replication engine to navigate complex log structures efficiently and consolidate distributed transactions into a single transaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data replication engines use multithreading to handle high transaction rates, then productivity is improved, but transaction events may be received out of order at the target system, worsening reliability
Solution Approach 1:
The patent applies preliminary action by pre-fetching change blocks from transaction-log disks and storing them in memory buffers before they are needed for replication. This allows the replication engine to process transactions in natural-flow order without waiting for disk I/O operations, thereby maintaining transaction order integrity while achieving high processing rates through multithreading.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of memory buffers that hold pre-fetched change blocks. These buffers act as intermediaries between the transaction-log disks and the replication engine, allowing multiple threads to access transaction data simultaneously without causing order violations, thus resolving the contradiction between multithreading productivity and transaction order reliability.
2Manufacturing precision
If the replication engine reads transaction logs directly from multiple disks, then manufacturing precision is improved, but device complexity increases due to navigating complex log structures
Solution Approach 1:
The patent applies copying by creating copies of change blocks from the transaction logs and storing them in memory buffers. Instead of directly navigating complex log structures on multiple disks, the replication engine works with copied data in memory, which simplifies the navigation process while maintaining data replication accuracy. This copying mechanism allows parallel access without the complexity of coordinating multiple disk reads.
3Productivity
If transaction events are processed in parallel threads, then productivity is improved, but materialization delays increase due to resynchronization requirements
Solution Approach 1:
The patent applies preliminary action by pre-fetching and buffering change blocks before they are needed for replication. This eliminates waiting time during the replication process, as threads can immediately process data from memory buffers rather than waiting for disk I/O operations. The resynchronization overhead is minimized because the natural-flow ordering is preserved in the buffer structure itself.
Data Source
AI summary
Methods and apparatus are provided for performing source-side merging of distributed transactions prior to replication, wherein a distributed transaction occurs at a plurality of nodes. A first node includes a database and an audit trail that stores database change events from the database of the first node. One or more other nodes each include a database and an audit trail that stores database change events from the database of the respective node. In use, a transaction is initiated which updates the database of the first node and the database at one or more of the other nodes. The database updates are captured in the audit trails of the respective nodes involved with the transaction. The first node receives and merges the database change events from the audit trails of each of the nodes involved in the transaction. The merged database change events are replicated via a replication engine only from the first node.