Dynamic Propagation Stream Splitting for Database Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database replication systems face challenges in efficiently managing changes across multiple databases, particularly in scenarios where new nodes are added or existing nodes become slow or fail, leading to performance impacts and data consistency issues.
Innovation Solution
The development of APIs and processes for configuring and reconfiguring multi-master information sharing topologies, including techniques for adding new nodes, promoting slaves to masters, splitting and merging propagation streams, and managing data streams to maintain performance and data integrity across database servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single propagation stream is used to distribute changes to multiple destination databases, then the system structure is simple, but performance degrades when destination databases are slow or fail
Solution Approach 1:
The patent divides a single propagation stream into multiple separate propagation streams, each dedicated to a specific destination database. This segmentation allows independent management and optimization of each stream, preventing slow or failed destinations from impacting the entire propagation system. The capture process can now serve multiple destinations through separate streams rather than a single shared stream.
2Productivity
If the propagation stream is split into multiple streams for each destination, then performance improves, but the system complexity increases
Solution Approach 1:
The patent implements dynamic stream configuration where propagation streams can be created, modified, and dropped based on runtime conditions. The system can dynamically determine whether to use a single stream or multiple streams based on destination performance, and can drop streams for failed destinations while maintaining streams for healthy destinations. This dynamic approach allows the system to adapt its complexity to actual performance needs.
3Reliability
If a destination database is slow in applying changes, then data consistency is maintained, but overall system performance is impacted
Solution Approach 1:
By segmenting the propagation system into separate streams for each destination, the patent isolates the performance impact of slow destinations to individual streams rather than affecting the entire system. Each destination database processes changes through its dedicated stream at its own pace, maintaining data consistency without blocking other destinations from receiving and applying changes efficiently.
4Adaptability or versatility
If new nodes are added to the replication environment, then system capacity increases, but configuration complexity and data consistency challenges arise
Solution Approach 1:
The patent enables dynamic addition of new destination databases to the replication environment. When a new destination is added, the system dynamically creates a new propagation stream from the capture process to the new destination. This dynamic configuration allows the system to scale capacity by adding nodes without requiring complex manual setup, as the propagation infrastructure adapts automatically to accommodate new participants in the replication topology.
Data Source
AI summary
Splitting and merging database object information sharing streams. Streams are also referred to herein as “propagations”. Splitting and merging information sharing streams can be used to improve performance in a information sharing environment when a failed or slow DBS impacts the performance. In one embodiment, an auto split process monitors the progress of applying changes at each node and detects the presence of a failed or a slow node. Once the failed or slow node is identified, the auto split process splits the propagation such that the offending node is sent through a separate propagation. Furthermore, an auto merge process can be started to monitor the newly created separate propagation. At a later point, the new propagation can be merged back into the original stream. For example, if the offending node catches up with other nodes, the auto merge process merges the newly created propagation back to the original propagation.