Parallel Scanner Threads for Database Log Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed database systems face performance bottlenecks and scalability issues due to the sequential nature of data replication, which limits the ability to process and distribute data efficiently across multiple replication paths, especially when priority transactions require immediate processing.
Innovation Solution
Implementing multiple scanner threads, each assigned to a specific replication path, allowing parallel processing and distribution of log records, thereby reducing performance bottlenecks and enabling immediate handling of priority transactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single scanner thread is used to scan the transaction log and distribute data to multiple replication paths, then the system structure is simple, but the replication throughput is limited and performance bottleneck occurs
Solution Approach 1:
The patent divides the single scanner thread into multiple scanner threads, with each thread responsible for scanning the transaction log for a specific replication path. This segmentation allows parallel processing of log records across multiple paths simultaneously, eliminating the sequential bottleneck and improving replication throughput without requiring complex coordination mechanisms.
Solution Approach 2:
The patent transitions from a single-dimensional sequential processing model to a multi-dimensional parallel processing model by introducing multiple scanner threads operating concurrently. Each thread operates in its own execution dimension, allowing simultaneous data distribution across multiple replication paths, thereby breaking the throughput limitation imposed by sequential processing.
2Productivity
If a single scanner thread processes data sequentially for multiple replication paths, then the system is easy to manage, but priority transactions cannot be processed immediately
Solution Approach 1:
The patent segments the transaction log scanning task by creating dedicated scanner threads for different replication paths, including priority paths. This allows priority transactions to be scanned and distributed immediately by their dedicated thread without waiting for lower-priority transactions to be processed, achieving immediate processing of urgent data while maintaining straightforward thread management.
3Adaptability or versatility
If the number of replication destinations and paths is increased, then the system versatility is improved, but the performance bottleneck is exacerbated
Solution Approach 1:
The patent creates a scalable architecture where each replication path has its own dedicated scanner thread, allowing the system to accommodate any number of replication destinations and paths. The universal scanner thread design can be dynamically added or removed based on replication requirements, providing flexibility while maintaining high throughput through parallel processing regardless of the number of paths.
Data Source
AI summary
Log based replication using parallel scanners is described. An example method includes establishing replication paths from a primary database to a plurality of destination servers. A plurality of scanner threads are spawned to scan a transaction log of the primary database. Specifically, one scanner thread is assigned to each replication path corresponding to each destination server. For each scanner thread, data bound to a replication path associated with the scanner thread is selected from a subset of the transaction log. Accordingly, the data is distributed to a destination server associated with the replication path.