Automatic Parallelism Tuning for Database Apply Processes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database replication systems face performance issues due to dependencies between transactions and contention, which hinder automatic scaling with the number of processes used to apply source database transactions to a target database.
Innovation Solution
Implementing an automatic parallelism tuning mechanism that dynamically adjusts the number of active apply processes based on throughput values, considering factors like CPU, memory, workload size, and contention, to optimize performance and scalability for parallel workload processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple concurrent processes are used to apply source database transactions in parallel, then productivity is improved, but device complexity increases due to dependencies between transactions and contention issues
Solution Approach 1:
The system dynamically adjusts the number of active apply processes based on measured throughput and system conditions. The apply coordinator monitors performance metrics and automatically increments or decrements the number of processes, transforming a static configuration into a dynamic adaptive system that optimizes parallelism while managing complexity
Solution Approach 2:
The system implements feedback loops where the apply coordinator continuously measures throughput values from apply processes and uses this information to adjust the number of active processes. This closed-loop control enables the system to self-tune parallelism levels, improving productivity while automatically managing the complexity of process coordination
2Productivity
If the number of apply processes is increased to improve throughput, then productivity is improved, but reliability decreases due to contention issues and transaction dependencies
Solution Approach 1:
The system uses dynamic process adjustment to match the number of apply processes with actual system capacity and workload characteristics. By measuring throughput and automatically adjusting process counts, the system avoids the reliability issues associated with excessive parallelism while maintaining optimal productivity
Solution Approach 2:
The apply coordinator autonomously manages the number of active apply processes by monitoring throughput and making adjustment decisions without external intervention. This self-service mechanism ensures that the system maintains reliable operation by automatically reducing process counts when contention or dependencies cause instability
Data Source
AI summary
Techniques are provided for automatic parallelism tuning. At least one batch of change records is assigned to one or more apply processes in a set of active apply processes. A first throughput value is periodically determined based on a number of processed change records in a first time interval. An increment adjustment is periodically performed, including adding an additional apply process, determining a second throughput value, and removing the additional apply process from the set of active apply processes if the second throughput value is not greater than a previous first throughput value by at least an increment threshold. A decrement adjustment is periodically performed, including removing an apply process, determining a third throughput value, and replacing the removed apply process in the set of active apply processes if the third throughput value is not greater than the previous first throughput value by at least a decrement threshold.