GPU Sharing Scheduler With Interference Prediction for Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing GPU sharing methods in cloud environments lead to inconsistent job completion times (JCT) and violate service level agreements (gSLA) due to unpredictable interference between distributed deep learning jobs, resulting in inefficient GPU utilization.
Innovation Solution
A scheduling method that profiles GPU resource usage and employs a deep learning-based interference prediction model to select job combinations that satisfy gSLA, using a TensorShare scheduler to mitigate performance degradation and improve GPU sharing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If GPU sharing is implemented to improve GPU utilization, then GPU utilization is improved, but job completion time becomes inconsistent and service level agreements are violated
Solution Approach 1:
The system performs preliminary action by predicting interference between distributed deep learning jobs before they are scheduled onto shared GPUs. The interference prediction model estimates the impact on job completion time, allowing the scheduler to make informed decisions about job placement. This preliminary prediction prevents gSLA violations before they occur, while still enabling GPU sharing to improve utilization.
2Productivity
If multiple distributed deep learning jobs share a GPU concurrently, then GPU utilization is improved, but training speed degrades due to resource usage interference
Solution Approach 1:
The system implements feedback by using the interference prediction model to continuously assess the impact of job combinations on training speed. The model provides feedback about expected performance degradation, allowing the scheduler to select job combinations that minimize interference. This feedback mechanism enables the system to maintain higher GPU utilization while preserving acceptable training speeds through intelligent job pairing.
3Loss of time
If GPU sharing is used to reduce job completion time, then GPU time is reduced, but interference prediction becomes more complex
Solution Approach 1:
The system introduces an intermediary element - the interference prediction model - that acts as a mediator between job scheduling decisions and actual GPU execution. This model simplifies the complex interference dynamics by providing predictable estimates of job completion time impacts. The intermediary model enables the scheduler to make rapid, informed decisions without directly simulating complex interference scenarios, thus reducing overall job completion time while managing prediction complexity.
Data Source
AI summary
Disclosed herein is an apparatus and method for interference prediction and guarantee of GPU sharing for distributed deep learning jobs. There is provided a scheduling method performed by a computing device, according to an embodiment. The scheduling method includes: receiving a distributed training job (DT job) from at least one user to register the DT job in a scheduling queue; generating candidate DT job combinations by filtering multiple DT job combinations, each consisting of one pre-scheduled first DT job in one of GPUs included in a GPU cluster and one of the DT jobs registered in the scheduling queue; and selecting a DT job to be executed concurrently with the first DT job in the one of the GPUs.