Job Scheduling Based on Execution History
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, especially those utilizing cloud resources, there is a challenge in managing job scheduling to prevent large-scale concurrent failures caused by flawed programs, which can lead to increased resource usage costs and inefficiencies due to the lack of effective trustworthiness assessment of job execution histories.
Innovation Solution
A job scheduling system that determines a resource allocation score based on job execution history, allowing for the initial portion of jobs to be scheduled while delaying the remainder, and dynamically adjusts resource allocation based on the success or failure of the initial job execution, thereby throttling resource usage and reducing the likelihood of mass failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all jobs in a workload are scheduled for concurrent execution, then productivity is improved, but reliability deteriorates due to potential mass failures from flawed programs
Solution Approach 1:
The system performs preliminary assessment of workload trustworthiness by analyzing execution history and resource allocation scores before full concurrent execution. This preliminary action allows the system to identify potentially flawed workloads and prevent mass failures while still enabling productive execution of trustworthy workloads.
Solution Approach 2:
The system schedules only a portion of jobs for concurrent execution based on the resource allocation score, rather than executing all jobs simultaneously. This partial action approach limits the scope of potential failures while maintaining productivity for the scheduled portion, and allows progressive execution of additional jobs as trustworthiness is confirmed.
2Reliability
If resource allocation is restricted to prevent failures, then reliability is improved, but productivity deteriorates due to delayed job execution
Solution Approach 1:
The system dynamically adjusts the resource allocation score and job scheduling decisions based on real-time execution outcomes and historical data. This dynamic approach allows the system to be conservative when reliability concerns are high and more aggressive when trustworthiness is confirmed, optimizing both reliability and productivity adaptively rather than using fixed restrictions.
Solution Approach 2:
The system uses feedback from job execution results to update resource allocation scores and adjust future scheduling decisions. Successful executions increase the resource allocation score, allowing more aggressive scheduling, while failures decrease the score, triggering more conservative resource allocation. This feedback loop ensures that productivity restrictions are relaxed as reliability is demonstrated.
3Reliability
If resource allocation score is used to control job scheduling, then reliability is improved, but device complexity increases due to additional scoring and monitoring mechanisms
Solution Approach 1:
The system uses the workload's own execution history and performance data to generate its resource allocation score, rather than requiring external manual assessment. This self-service approach automatically builds trustworthiness metrics from observed behavior, reducing the need for complex external verification mechanisms while improving reliability through data-driven decision making.
Data Source
AI summary
Methods, systems, and computer-readable media for job scheduling based on job execution history are disclosed. A request is received to schedule a workload comprising a plurality of jobs. A resource allocation score for the workload is determined. The resource allocation score represents (at least in part) an estimated likelihood of successful execution of the workload. A first portion of the workload is scheduled for execution, and a remaining portion (if any) of the workload is delayed. A quantity of jobs in the first portion of the workload is determined based (at least in part) on the resource allocation score. Execution of the first portion of the workload is initiated.