GPU Core Packing for Shared Inference Job Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU schedulers fail to maximize utilization and lead to resource waste due to inefficient job packing, resulting in out-of-memory errors and suboptimal throughput for inference jobs.
Innovation Solution
A computer system and method for job packing that sorts computational nodes into groups based on GPU cores, employing dynamic memory and core management to optimize resource utilization, allowing concurrent loading and scheduling across multiple dimensions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple jobs are dispatched into one graphical processing unit card, then job throughput is improved, but out-of-memory errors occur
Solution Approach 1:
The patent segments jobs into different priority levels (high priority and low priority jobs) and divides the GPU core allocation accordingly. High priority jobs receive guaranteed core resources while low priority jobs use remaining resources, preventing memory exhaustion and out-of-memory errors while maintaining high throughput
Solution Approach 2:
The patent implements dynamic core allocation where the number of cores assigned to each job changes based on system state. The scheduler dynamically adjusts core distribution between concurrent jobs based on memory availability, job priority, and current utilization, preventing memory overflow while maximizing throughput
2Ease of operation
If time-slice scheduling treats a whole graphical processing unit card as minimal resource unit, then scheduling simplicity is improved, but graphical processing unit resource waste occurs
Solution Approach 1:
The patent segments the GPU card into individual core units that can be independently allocated to different jobs. Instead of treating the entire GPU card as a single resource unit, the scheduler can allocate specific numbers of cores to different jobs, enabling fine-grained resource utilization and eliminating GPU resource waste
Solution Approach 2:
The patent introduces a new dimension of resource allocation by dividing the GPU card into discrete core units along the resource dimension. This transforms the scheduling problem from allocating entire GPU cards to allocating specific core counts, enabling more flexible and efficient resource utilization
3Reliability
If one inference job is dispatched per graphical processing unit card, then memory stability is improved, but GPU utilization efficiency deteriorates
Solution Approach 1:
The patent merges multiple inference jobs into a single GPU card by allocating multiple cores from the same GPU to different jobs simultaneously. High priority and low priority jobs run concurrently on the same GPU card with dynamically allocated core resources, improving GPU utilization efficiency while maintaining memory stability through controlled resource distribution
Data Source
AI summary
A computer implemented method controls in a core dimension packing the plurality of processing unit cores to reduce partial use of the plurality of processing unit cores by concurrently loading into the plurality of processing unit cores: all computational nodes of at least one member selected from the group consisting of the first number of whole groups and the first number of partial groups; and all computation nodes of at least one member selected from the group consisting of the second number of whole groups and the second number of partial groups. According to other illustrative embodiments, a computer system and a computer program product for packing the plurality of processing cores are provided. As a result, the illustrative embodiments can provide a technical effect of increasing the performance in running jobs in a computer system in which partial use of processing unit cores can be reduced.