Multi-threaded Replication Engine Transaction Serialization

Solution Overview

Problem

Asynchronous data replication in multi-threaded systems often leads to database corruption and inconsistency due to the lack of control over the order of transactions and changes across multiple paths, resulting in data collisions and referential integrity violations.

Innovation Solution

Implementing multi-threaded architectures with mechanisms to serialize and re-serialize transaction data across multiple paths, ensuring that changes are applied in natural flow order and maintaining referential integrity through techniques like Expected Ends algorithms, Transaction Serializers, and coordinated commit protocols.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multi-threaded replication is used to increase processing capacity, then productivity is improved, but database consistency and referential integrity deteriorate due to uncontrolled transaction ordering across multiple paths

Engineering Contradiction:

Improveprocessing capacityVSAvoiddatabase consistency

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The replication engine segments transactions into smaller units and distributes them across multiple threads and paths. Each thread processes a subset of transactions independently, increasing parallelism and processing capacity while maintaining consistency through controlled segmentation of the replication workload

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary actions by capturing transaction information before execution and storing it in a change queue. This allows the replication engine to review and verify transaction details before applying them to the target database, ensuring referential integrity is maintained even when transactions are processed in parallel across multiple threads

Inventive Principle:

Principle #10Preliminary action

2Productivity

If asynchronous replication is used to improve system availability, then productivity is improved, but data consistency deteriorates due to lack of control over transaction order across multiple paths

Engineering Contradiction:

Improvesystem availabilityVSAvoiddata consistency

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The replication engine implements feedback mechanisms where the target database sends acknowledgments back to the source database about transaction processing status. This feedback loop allows the system to track which transactions have been successfully replicated and maintain a consistent state across all database copies, enabling asynchronous replication with guaranteed data consistency

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system captures and queues transaction information before it is applied to the target database. This preliminary capture allows the replication engine to review transactions, verify their validity, and ensure they are applied in the correct sequence, maintaining data consistency even in asynchronous distributed environments

Inventive Principle:

Principle #10Preliminary action

3Productivity

If multiple replication paths are used to increase throughput, then productivity is improved, but referential integrity deteriorates due to data collisions and inconsistent transaction ordering

Engineering Contradiction:

ImprovethroughputVSAvoidreferential integrity

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The replication engine segments transactions into smaller units and distributes them across multiple threads and paths. Each thread processes a subset of transactions independently, increasing parallelism and processing capacity while maintaining consistency through controlled segmentation of the replication workload

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The replication engine implements feedback mechanisms where the target database sends acknowledgments back to the source database about transaction processing status. This feedback loop allows the system to track which transactions have been successfully replicated and maintain a consistent state across all database copies, enabling asynchronous replication with guaranteed data consistency

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS7801851B2Method for ensuring referential integrity in multi-threaded replication engines

Publication Date: 2010.09.21 INTEL CORP

AI summary

During replication of transaction data from a source database to a target database via a change queue associated with the source database, one or more multiple paths are provided between the change queue and the target database. The one or more multiple paths cause at least some of the transaction data to become unserialized. At least some of the unserialized data is reserialized prior to or upon applying the originally unserialized transaction data to the target database. If the current transaction load is close or equal to the maximum transaction load capacity of a path between the change queue and the target database, another path is provided. If the maximum transaction threshold limit of an applier associated with the target database has been reached, open transactions may be prematurely committed.