Predictive Job Scheduling for Backup Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy scheduling techniques for computing jobs, such as backup jobs, fail to accurately predict resource requirements and are inadequate in dynamically adjusting to changing resource availabilities, leading to potential overloading and failure to meet service level agreements (SLAs).
Innovation Solution
The implementation of a predictive model that continuously reassesses system resources and prioritizes backup jobs by using time-series representations of resource usage, confidence intervals, and dynamic scheduling techniques to adjust resource allocations based on changing conditions, ensuring that backup jobs are scheduled efficiently without impacting foreground jobs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If backup jobs are scheduled using legacy techniques (just-in-time scheduling, earliest start scheduling), then scheduling simplicity is maintained, but prediction accuracy deteriorates leading to frequent scheduling errors
Solution Approach 1:
The system performs preliminary actions by continuously collecting resource usage measurements and training prediction models in advance, so that accurate predictions are ready when scheduling decisions are needed. This resolves the contradiction by preparing prediction data beforehand, maintaining scheduling simplicity while improving accuracy.
Solution Approach 2:
The system implements feedback mechanisms where actual resource usage is continuously measured and fed back to update prediction models. This feedback loop improves prediction accuracy over time while keeping the scheduling interface simple, as the system automatically learns from past performance without requiring complex manual intervention.
2Stability of the object's composition
If resource allocations are statically determined, then scheduling stability is maintained, but adaptability to changing resource availabilities deteriorates
Solution Approach 1:
The system applies dynamics by making resource allocations adaptive rather than static. Prediction models continuously update based on measured resource usage, allowing the system to dynamically adjust allocations while maintaining stability through systematic update procedures. This resolves the contradiction by enabling the system to adapt to changing conditions while preserving scheduling stability.
Solution Approach 2:
The system performs self-service by automatically measuring resource usage, updating prediction models, and adjusting schedules without external intervention. This self-adjusting mechanism maintains stability through consistent behavior while improving adaptability, as the system autonomously responds to changing resource availabilities.
3Productivity
If multiple backup jobs are scheduled concurrently, then productivity increases, but system reliability deteriorates due to potential overloading and SLA violations
Solution Approach 1:
The system changes parameters by using prediction models to estimate completion times and resource requirements for multiple concurrent jobs. Based on these predictions, the system adjusts scheduling parameters to ensure SLA compliance while maximizing throughput. This resolves the contradiction by enabling safe concurrency through informed parameter adjustment.
Solution Approach 2:
The system performs preliminary prediction and assessment before scheduling multiple backup jobs concurrently. By predicting resource availability and job completion times in advance, the system can confidently schedule multiple jobs while maintaining SLA compliance, thus improving productivity without sacrificing reliability.
4Loss of time
If predictions are made far in advance, then scheduling lead time increases, but prediction accuracy deteriorates due to unforeseen events
Solution Approach 1:
The system maintains continuity by continuously collecting resource usage measurements and continuously training prediction models. This ongoing process allows the system to make predictions at optimal intervals, balancing lead time and accuracy. The continuous measurement and update cycle ensures predictions remain accurate while providing sufficient scheduling lead time.
Solution Approach 2:
The system employs periodic action by scheduling prediction updates and model training at regular intervals based on measured resource usage patterns. This periodic reinforcement of predictions maintains accuracy over time while providing consistent scheduling lead time, resolving the contradiction between advance scheduling and prediction reliability.
Data Source
AI summary
Measurements comprising time-series stimuli and time-series responses of a computing platform that has executed a first set of jobs are collected over a first time period. The measurements are used to form a query-able predictive model pertaining to resource usage demand predictions for the first set of jobs. A second set of job records describe a second set of jobs to be invoked in a second time period. The predictive model is queried to determine a likelihood to complete by the predicted finish time based on resource usage demand predictions for the first set of jobs. A weighting factor related to a likelihood to complete the second set of jobs by a particular time is calculated. A reward value based on the weighting factor is assigned to respective jobs in the second set of jobs. Some of the second set of jobs are rescheduled, based on a then-current reward value.