Job Resource Estimation via Historical Execution Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing grids, users face a dilemma when submitting jobs as they must guess the required resources, leading to potential delays or resource wastage due to over or underestimation, with existing methods lacking an efficient solution for determining the necessary computing resources.
Innovation Solution
A mechanism that automatically estimates computing resources needed for a job by analyzing job characteristics and accessing a database of previously executed jobs' resource usage information to derive an accurate resource estimate, relieving users of the need to guess resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the user overestimates the resources needed, then the job can be executed without resource shortages, but the job execution is delayed and grid efficiency is reduced
Solution Approach 1:
The system performs preliminary resource estimation by analyzing historical execution data and job characteristics before the job is submitted to the queue. This advance preparation allows the DRM to allocate resources more accurately without requiring the user to overestimate, thereby reducing waiting time while ensuring sufficient resources are available for execution.
Solution Approach 2:
The system utilizes feedback from historical job executions stored in the database, analyzing actual resource consumption patterns of similar jobs. This feedback mechanism enables continuous improvement of resource estimation accuracy, allowing the system to learn from past performances and provide increasingly precise predictions for future job resource allocation.
2Reliability
If the user overestimates the resources needed, then the job can be executed without resource shortages, but resources are wasted and grid efficiency is reduced
Solution Approach 1:
The system performs preliminary resource estimation by analyzing historical execution data and job characteristics before the job is submitted to the queue. This advance preparation allows the DRM to allocate resources more accurately without requiring the user to overestimate, thereby reducing waiting time while ensuring sufficient resources are available for execution.
Solution Approach 2:
The system utilizes feedback from historical job executions stored in the database, analyzing actual resource consumption patterns of similar jobs. This feedback mechanism enables continuous improvement of resource estimation accuracy, allowing the system to learn from past performances and provide increasingly precise predictions for future job resource allocation.
3Productivity
If the user underestimates the resources needed, then resources are allocated efficiently, but the job may be killed midstream
Solution Approach 1:
The system performs preliminary resource estimation by analyzing historical execution data and job characteristics before the job is submitted to the queue. This advance preparation allows the DRM to allocate resources more accurately without requiring the user to overestimate, thereby reducing waiting time while ensuring sufficient resources are available for execution.
Solution Approach 2:
The system utilizes feedback from historical job executions stored in the database, analyzing actual resource consumption patterns of similar jobs. This feedback mechanism enables continuous improvement of resource estimation accuracy, allowing the system to learn from past performances and provide increasingly precise predictions for future job resource allocation.
4Ease of operation
If the user is required to determine resources needed, then the user has control over resource allocation, but the user lacks knowledge to make accurate estimates
Solution Approach 1:
The system enables self-service by automatically performing resource estimation based on job characteristics and historical data, eliminating the need for users to manually determine resource requirements. The DRM leverages its access to historical execution information to provide accurate estimates, freeing users from this complex task while maintaining their ability to review and adjust estimates if needed.
Solution Approach 2:
The system introduces an intermediary resource estimation mechanism between the user and the DRM. This intermediary analyzes job characteristics and historical data to bridge the gap between user intent and actual resource requirements, providing recommended allocations that users can accept or modify without needing deep technical knowledge of resource planning.
Data Source
AI summary
A mechanism is provided for estimating the computing resources needed to execute a job. The mechanism receives a request to execute a new job. The mechanism processes the request to determine a set of job characteristics for the new job, and accesses a database containing execution information for a plurality of previously executed jobs. The mechanism obtains from the database a set of execution information associated with a particular previously executed job having similar or identical job characteristics as the new job. From this execution information, the mechanism determines what computing resources were actually used to execute the particular previously executed job. Then, based upon this information, the mechanism derives an estimate of the computing resources needed to execute the new job. By estimating the needed resources in this manner, the mechanism relieves the user of having to guess at what resources will be needed to execute a job.