HPC Scheduling Node Predicts Job Resource Consumption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing (HPC) systems face low resource utilization due to the mismatch between the peak resource requirements specified by users and the actual resources consumed during job execution, leading to inefficient job processing.
Innovation Solution
A scheduling node predicts resource consumption based on job categories, using historical data to estimate the actual resource usage, thereby scheduling jobs more accurately and improving resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the scheduling node allocates resources based on the peak resource quantity specified by the user, then the job can be guaranteed to have sufficient resources, but the resource utilization of the computing node becomes low
Solution Approach 1:
The system performs preliminary classification of jobs into different categories based on their characteristics before resource allocation. This preliminary action enables the scheduling node to predict resource consumption patterns in advance, allowing for more accurate resource allocation that balances guarantee and utilization
Solution Approach 2:
The system changes the parameter used for resource allocation from fixed peak resource quantity to dynamic predicted resource quantity based on job category. This parameter change allows the scheduling node to adapt resource allocation to actual job needs, improving both reliability and productivity
2Ease of operation
If the scheduling node uses the maximum resource quantity specified by the user for allocation, then resource allocation is simple and straightforward, but the actual resource consumption is not fully utilized
Solution Approach 1:
The system performs preliminary classification of jobs into different categories based on their characteristics before resource allocation. This preliminary action enables the scheduling node to predict resource consumption patterns in advance, allowing for more accurate resource allocation that balances guarantee and utilization
Solution Approach 2:
The system enables jobs to effectively specify their own resource needs through category-based prediction. Instead of requiring users to manually specify peak resources, the system automatically determines appropriate resource allocation based on job category and historical data, making the process more efficient
Data Source
AI summary
A job processing method is provided. In this method, after receiving a to-be-processed job, a scheduling node in a high-performance computing system predicts, based on a category of the job, a quantity of resources consumed in real time in a process of executing the job. Because quantities of resources consumed in real time in a process of executing jobs of a same category are close, compared with a specified resource quantity, a predicted resource quantity is closer to a quantity of resources actually consumed by the job.