Remote Database Change Data Capture via Ordered Log Block Replication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing change data capture systems face challenges in efficiently replicating and maintaining data consistency across geographically remote databases, particularly in maintaining the order of transactions and handling lost or delayed messages during data transmission.

Innovation Solution

A system and method for replicating a source database to a remote database, where source log blocks containing database transactions are transmitted over a data transmission system, and received components extract and reorder transaction data to ensure accurate replication, using middleware messaging and unique serial numbers to manage and reassemble log blocks, ensuring timely and ordered data application.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If source log blocks are transmitted over a data transmission system to a remote location, then data replication across remote locations is achieved, but message loss or delay may occur during transmission

Engineering Contradiction:

Improvedata replication reliabilityVSAvoidmessage loss or delay

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The patent applies preliminary action by assigning unique serial numbers to source log blocks before transmission. This pre-numbering allows the target system to detect missing or delayed messages and request retransmission, ensuring reliable data replication even over unreliable networks.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements feedback mechanisms where the target database system monitors received log blocks, identifies gaps in the sequence, and sends requests for missing log blocks back to the source system. This feedback loop ensures complete and accurate data replication despite transmission issues.

Inventive Principle:

Principle #23Feedback

2Productivity

If source log blocks containing multiple database transactions are transmitted together, then transmission efficiency is improved, but maintaining transaction order becomes more difficult

Engineering Contradiction:

Improvetransmission efficiencyVSAvoidtransaction order maintenance

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The patent segments transactions into individual log blocks while maintaining their sequential order within each block. Each log block is assigned a unique serial number, allowing the target system to reassemble transactions in the correct order even when blocks arrive out of sequence, thus maintaining both efficiency and precision.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary logging mechanism that buffers and orders log blocks at the target system before applying transactions. This intermediary layer decouples the transmission process from the transaction application process, allowing efficient batch transmission while ensuring precise order maintenance through sequential processing.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Manufacturing precision

If log blocks are buffered and processed in order at the target location, then transaction order is maintained, but processing delay increases

Engineering Contradiction:

Improvetransaction order maintenanceVSAvoidprocessing delay

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent applies partial action by processing log blocks as they arrive in sequence rather than waiting for complete batches. The target system applies transactions from received log blocks immediately in order, rather than buffering all blocks, thus reducing delay while maintaining order through sequential processing of available data.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS9940347B2Systems and methods for high speed remote change data capture

Publication Date: 2018.04.10 IGNITE ENTERPRISE SOFTWARE SOLUTIONS INC

AI summary

A system and method are described for change data capture wherein a source database is replicated to a remote database. In one embodiment, a data storage device contains source log blocks containing information regarding database transactions processed for a source database, wherein the data storage device and source database are located in a first location, and at least some of the source log blocks include a plurality of database transactions in the same order as processed in the source database. Messages containing source log blocks are sent from the first location to a second location geographically remote from the first location. One or more components at the second location receive source log blocks from the messages, extract the source transaction data from the source log blocks, and use the source transaction data to write database transaction information to a second database based on the extracted source transaction data.