Adaptive Backup Host Throttling for Metadata Fetch Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Frequent backups of enterprise resource planning (ERP) systems and customer resource management (CRM) systems can take a prohibitive amount of time and burden the production system with competing resource demands.
Innovation Solution
Implementing adaptive throttling in a universal backup host that adjusts the thread pool size based on latency and request aggregation, allowing for efficient metadata fetching and backup operations by resizing the thread pool in response to latency and request patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If frequent backups are performed on production systems, then data recovery capability is improved, but system performance and resource availability deteriorate
Solution Approach 1:
The backup system dynamically adjusts the thread pool size based on real-time latency measurements and request aggregation patterns. The system monitors latency metrics and automatically resizes the thread pool to optimize backup performance while minimizing impact on production system resources, transforming a static resource allocation into a dynamic adaptive mechanism.
Solution Approach 2:
The system changes the parameter of thread pool size based on observed latency patterns and request rates. By monitoring system response times and adjusting the number of concurrent backup threads accordingly, the system adapts resource allocation to current load conditions, resolving the contradiction between backup frequency and system performance.
2Speed
If a large thread pool is used for backup operations, then backup speed is improved, but resource competition and system burden increase
Solution Approach 1:
The backup system implements feedback mechanisms by continuously measuring latency metrics and using this information to adjust thread pool size. The system aggregates latency samples over time and uses this feedback to dynamically resize resources, ensuring backup operations proceed at optimal speed without creating excessive resource competition.
Solution Approach 2:
The backup system performs self-adjustment by automatically monitoring its own performance metrics and resizing the thread pool without external intervention. The system services itself by detecting latency patterns and autonomously optimizing resource allocation, balancing backup speed with system resource availability.
Data Source
AI summary
Systems and methods to throttle a universal backup host are described. The system executes a job, at a backup host, to back up a file set from a source host including fetching metadata from the source host. The system identifies a first operation set from operation sets, the operation set including a first operation. The system communicates, in parallel, requests for metadata items, over a network, to the source host, receives responses, and processes the responses by utilizing threads from a thread pool. The system generates latencies, counts the number of requests, and stores the latencies and number of requests in samples. The system aggregates the samples responsive to a timeout. The system resizes the thread pool based on the aggregating. Finally, the system backs up the file set from the source host based on the metadata.