Job Scheduling System for Resource-Intensive Cloud Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based job execution environments, resource-intensive computing jobs can cause resource abuse, leading to overloaded queues, delayed jobs, and execution failures, requiring manual intervention that is time-consuming and error-prone.
Innovation Solution
A method that predicts the resource intensity of computing jobs based on historical job data and features of the job, allowing for preemptive identification and assignment of resource-intensive jobs to appropriate job execution resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual reassignment of resource-intensive jobs is implemented, then resource abuse is reduced and queue overload is prevented, but operational time increases and error rate increases
Solution Approach 1:
The system performs preliminary classification of computing jobs as resource-intensive or non-resource-intensive before they are assigned to job executors. This advance classification enables automatic routing decisions, eliminating the need for manual reassignment while maintaining reliable job execution by preventing resource abuse upfront
Solution Approach 2:
The job classification system automatically identifies and routes resource-intensive jobs without requiring manual intervention. The system serves itself by using the classification information to autonomously make assignment decisions, thereby reducing both operational time and human error
2Productivity
If resource-intensive jobs are allowed to execute freely, then job submission flexibility is maintained, but queue overload occurs and other jobs are delayed
Solution Approach 1:
The system applies different handling strategies to different types of jobs based on their classification. Resource-intensive jobs are routed to specialized executors or handled with specific policies, while non-resource-intensive jobs follow standard processing. This localized quality approach maintains overall productivity while preventing delays to individual jobs
3Extent of automation
If classification of computing jobs is implemented, then automatic assignment is enabled and manual intervention is reduced, but system complexity increases
Solution Approach 1:
The classification system divides jobs into distinct categories (resource-intensive and non-resource-intensive) based on simple criteria. This segmentation enables straightforward automatic assignment logic without requiring complex decision-making systems, thereby maintaining high automation while controlling system complexity
Data Source
AI summary
Systems and methods described herein relate to the handling of resource-intensive computing jobs in a cloud-based job execution environment. An unexecuted computing job has a plurality of features. A resource intensity prediction is generated for the unexecuted computing job based on the features and on historical job data that classifies each of a plurality of executed computing jobs as either resource intensive or non-resource intensive. The resource intensity prediction indicates that the unexecuted computing job is predicted to be classified as resource intensive. A predicted resource intensity category of the unexecuted computing job is determined. Utilization data associated with one or more of a plurality of job execution destinations may be accessed. The unexecuted computing job may be assigned to a selected job execution destination from among the plurality of job execution destinations based on the predicted resource intensity category and the utilization data.