IO Sizing Throttling for Backup Server Concurrency Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Backup servers face contention and performance issues due to high concurrency of operations, leading to potential hangs or timeouts, especially in busy environments where processor resources are overloaded, causing IO bottlenecks and starvation.
Innovation Solution
Implementing IO sizing throttling to classify and prioritize backup operations based on their IO requirements, allowing the backup server to manage concurrent operations within the available IOPS limits of the underlying infrastructure, thereby preventing system unresponsiveness and ensuring smooth operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If high concurrency of backup operations is allowed, then productivity is improved, but system reliability deteriorates due to contention and timeouts
Solution Approach 1:
The system dynamically changes the concurrency parameter by adjusting the number of allowed parallel backup operations based on current system load and IO capacity. When IO bottlenecks are detected, the system reduces the concurrency level to prevent timeouts, while allowing higher concurrency when resources are available, thus resolving the contradiction between productivity and reliability
Solution Approach 2:
The system implements feedback mechanisms by monitoring backup operation performance metrics such as completion status, timeout occurrences, and IO wait times. This feedback is used to dynamically adjust the concurrency control policy, reducing operations when timeouts occur and increasing them when operations complete successfully, thereby maintaining both high productivity and reliability
2Productivity
If more concurrent operations are permitted, then productivity increases, but device complexity increases due to contentions on shared resources
Solution Approach 1:
The system introduces an intermediary component that acts as a gatekeeper for backup operations. This intermediary monitors resource availability and mediates access to shared catalog database resources, controlling the flow of concurrent operations to avoid direct contention. By inserting this intermediary layer, the system can maintain high productivity while simplifying the management of resource contentions
Solution Approach 2:
The system segments backup operations into different priority levels and categories (e.g., catalog maintenance vs. actual backups). By segmenting operations, the system can manage them in organized batches with controlled concurrency for each segment, reducing the overall complexity of managing all operations simultaneously while maintaining high throughput
3Productivity
If IO operations are increased to handle more backups, then productivity improves, but loss of time increases due to IO bottlenecks and starvation
Solution Approach 1:
The system implements periodic action by batching backup operations and processing them in controlled intervals rather than continuously. This periodic approach allows the system to accumulate multiple backup requests and execute them in organized waves, ensuring that IO resources are utilized efficiently without causing starvation, thereby improving productivity while minimizing wait time losses
Data Source
AI summary
Methods and apparatuses select service tasks according to allocations of an available usage rate of a common processing resource are described. An ordering relationship is updated among the service tasks. E service task can have a rate of usage of the common processing resource. Each service may be associated with one of multiple task types. An allocation of the available rate of usage may be determined among the task types. The allocation can indicate a portion of the available rate of usage for each task type. The service tasks may be selected according to the allocation and the ordering relationship. At least one of the selected service tasks is associated with each task type associated with the service tasks.