Job Preemption in High Performance Computing Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing clusters face inefficiencies in processing jobs due to long completion times and resource allocation imbalances, where high priority jobs may not be allocated sufficient resources promptly, leading to delayed completion.
Innovation Solution
A scheduling system that identifies preemptor jobs, both queued and running, capable of utilizing more resources than allocated, allowing these jobs to preempt other running jobs to free resources for higher priority tasks, ensuring efficient allocation and completion of high priority jobs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional schedulers only allow queued jobs to preempt running jobs, then resource allocation follows strict queue order, but high priority running jobs cannot dynamically acquire additional resources, leading to suboptimal resource utilization and delayed completion of high priority tasks
Solution Approach 1:
The patent implements dynamic preemption where running jobs can preempt other running jobs based on priority and resource availability. The system continuously evaluates job priorities and resource allocations, allowing high priority running jobs to dynamically acquire additional resources from lower priority running jobs, rather than following static queue-based allocation
Solution Approach 2:
The system changes the preemption parameter from binary (queued vs running) to multi-state (queued jobs can preempt running jobs, and running jobs can preempt other running jobs). This parameter expansion enables flexible resource reallocation where any job can preempt any other job based on priority levels and resource needs, resolving the contradiction between productivity improvement and system complexity
2Reliability
If resources are statically allocated to running jobs, then resource allocation is simple and stable, but high priority jobs cannot obtain additional resources during execution, leading to delayed completion
Solution Approach 1:
The system performs preliminary evaluation of job priorities and resource requirements before preemption occurs. The scheduler identifies potential preemptor jobs and their resource needs in advance, allowing high priority jobs to quickly acquire necessary resources when opportunities arise, reducing completion time while maintaining allocation stability through structured evaluation criteria
Solution Approach 2:
The patent implements continuous feedback loops where the scheduler monitors running jobs, resource availability, and job priorities in real-time. This feedback mechanism allows the system to dynamically adjust resource allocations, preempting lower priority jobs when high priority jobs need additional resources, thereby reducing completion time while maintaining overall system stability through controlled, monitored changes
3Ease of operation
If the scheduler performs frequent scheduling passes to evaluate preemption opportunities, then resource allocation responsiveness improves, but system overhead and processing time increase
Solution Approach 1:
The system implements periodic scheduling passes at optimized intervals rather than continuous evaluation. The scheduler performs preemption evaluations at regular, strategically chosen moments, balancing responsiveness to job priority changes with reduced processing overhead. This periodic approach maintains allocation responsiveness while controlling the energy and computational costs of frequent scheduling operations
Data Source
AI summary
The preemption of running jobs by other running or queued jobs in a system that has processing resources. The system has running jobs, and queued jobs that are awaiting processing by the system. In a scheduling operation, preemptor jobs are identified, the preemptor jobs being jobs that are candidates for preempting one or more of the running jobs. The preemptor jobs include queued jobs, as well as running jobs that are capable of using more processing resource of the system. One of the other running jobs is preempted to free processing resources for the running job that was identified as a preemptor job. Accordingly, not only may queued jobs preempt running jobs, but currently running jobs may preempt other currently running jobs.