Streaming Analytics Component for Database Replication Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Change data replication systems face performance deficits due to the resource-intensive processing of long lists of row-level modifications, which create throughput bottlenecks in real-world data replication systems.
Innovation Solution
A computer-implemented method using a streaming analytics component identifies row-level modifications affecting common columns, generates a reconstructed modification statement, and sends it to an apply component, optimizing the application process by reducing the load on the target database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If row-level modifications are processed individually in change data replication systems, then data replication accuracy is maintained, but system throughput and performance deteriorate due to resource-intensive processing of long lists of modifications
Solution Approach 1:
The patent merges multiple individual row-level modification statements into a single batched modification statement. The system identifies common columns across multiple row modifications and combines them into one consolidated SQL statement with a composite WHERE clause, reducing the number of individual statements from N to 1 while preserving data accuracy through systematic column matching and value verification.
Solution Approach 2:
The patent segments the modification processing into distinct phases: analysis phase (identifying common columns and values across row modifications), reconstruction phase (building the batched modification statement), and validation phase (testing equivalence before applying). This segmentation allows optimization at each stage while maintaining overall data replication fidelity.
2Reliability
If multiple row-level modifications are applied to the target database, then data synchronization is achieved, but resource consumption and processing time increase creating throughput bottlenecks
Solution Approach 1:
The system performs preliminary analysis of the change stream to identify patterns of common columns and values before generating modification statements. By pre-processing the change stream to group modifications by affected columns and their values, the system prepares optimized batch statements in advance, reducing actual application time while ensuring data synchronization accuracy.
3Manufacturing precision
If individual row-level modification statements are sent to the target database, then precise data replication is maintained, but the load on the target database increases reducing overall system efficiency
Solution Approach 1:
The patent merges multiple individual row-level modification statements into a single batched modification statement. The system identifies common columns across multiple row modifications and combines them into one consolidated SQL statement with a composite WHERE clause, reducing the number of individual statements from N to 1 while preserving data accuracy through systematic column matching and value verification.
Data Source
AI summary
A computer-implemented method includes, by a streaming analytics component, identifying a source database table and a target database table. The target database table includes one or more target database rows and one or more target database columns. The method further includes identifying a change stream including a plurality of row-level modifications that cause the target database table to replicate the source database table. The method further includes determining that each row-level modification affects one or more common columns of the target database columns, wherein the common columns exhibit one or more common values for those of the target database rows that are affected by the row-level modifications. The method further includes generating, based on the common values and the common columns, a reconstructed modification statement and sending the reconstructed modification statement to an apply component. A corresponding computer program product and computer system are also disclosed.