Data Replication Monitoring with Streamed Event Metadata

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data replication systems lack real-time monitoring mechanisms to detect data inconsistencies and losses during replication across distant databases, leading to significant costs and latencies in identifying and addressing issues, which can have catastrophic consequences.

Innovation Solution

A method that uses a secondary change data capture mechanism to monitor transactions between source and target databases, creating transaction and replication event streams with metadata to identify and log any uncommitted transactions within a specified time threshold, providing immediate notification and root-cause analysis for inconsistencies.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual audits are used to detect data loss, then data consistency can be verified, but the cost is very high and significant time is lost

Engineering Contradiction:

Improvedata consistency verificationVSAvoidtime to detect data loss

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces manual audit mechanisms with an automated stream processing system that continuously monitors replication events. The system uses event streams from Change Data Capture (CDC) mechanisms to automatically track and verify data replication in real-time, eliminating the need for expensive manual audits while providing immediate detection of data loss.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent implements continuous monitoring of data replication through persistent event streams that track transactions from source to target databases in real-time. This continuous action allows immediate detection of replication failures and data loss, rather than waiting for periodic manual audits or batch processes.

Inventive Principle:

Principle #20Continuity of useful action

2Reliability

If batch mode operations are used to verify replication, then data issues can be detected, but problems are only spotted hours or days after they occur

Engineering Contradiction:

Improvereplication verificationVSAvoidlatency in detecting inconsistencies

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces batch mode verification operations with continuous stream processing that monitors replication events in real-time. The system processes replication events as they occur, providing immediate detection of inconsistencies rather than waiting for periodic batch operations.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system maintains continuous monitoring through persistent event streams that track replication status without interruption. This allows immediate detection and alerting of replication failures, eliminating the hours or days latency inherent in batch mode operations.

Inventive Principle:

Principle #20Continuity of useful action

3Device complexity

If no monitoring mechanism is implemented, then system complexity is reduced, but data loss cannot be detected until manual intervention

Engineering Contradiction:

Improvemonitoring system complexityVSAvoiddata loss detection capability

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent introduces event streams as intermediary components that capture and transmit replication events without adding significant system complexity. These streams act as mediators between the source database, replication coordinator, and target database, providing monitoring capability through a lightweight infrastructure built on existing CDC mechanisms.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback loops where replication events are continuously monitored and compared between source and target. When inconsistencies are detected, the system generates alerts and notifications, providing immediate feedback about data loss or replication failures without requiring complex monitoring infrastructure.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS10409807B1Apparatus and method for data replication monitoring with streamed data updates

Publication Date: 2019.09.10 STRIIM INC

AI summary

A method implemented in a computer network includes identifying a transactional change data capture event at a transactional database. A transaction event stream is created with metadata characterizing the transactional change data capture event. A replication change data capture event is identified at a replication database corresponding to the transactional database. A replication event stream with metadata characterizing the replication change data capture event is created. The transaction event stream and the replication event stream are evaluated to selectively identify a replication performance failure within a specified time threshold of the replication performance failure.