Microservice-Based Training System for Heterogeneous GPU Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current microservice systems face challenges in efficiently training deep neural networks due to resource monopolization by individual training jobs, complexity in managing heterogeneous GPUs, and reduced performance from straggler issues in data parallelization.
Innovation Solution
A microservice-based training system utilizing a heterogeneous GPU cluster, where training jobs are partitioned into microservices and dynamically scheduled across heterogeneous processing units, enabling efficient distributed processing and resource management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a GPU is allocated to a training job, then the training job can execute, but the GPU is monopolized for a long time preventing other jobs from using it
Solution Approach 1:
The patent segments a monolithic training job into multiple microservices that can be independently scheduled and executed on different GPUs. Each microservice represents a smaller unit of work that can be dynamically allocated, preventing any single job from monopolizing all GPUs for an extended period.
Solution Approach 2:
The system dynamically schedules microservices based on current GPU availability and job requirements. Rather than static allocation, the scheduler can reassign microservices to different GPUs as jobs complete or new resources become available, optimizing overall system productivity.
2Productivity
If multiple training jobs are executed simultaneously in a heterogeneous cluster, then system throughput increases, but the determination of job allocation becomes more complicated
Solution Approach 1:
By dividing training jobs into standardized microservices, the system creates uniform units that are easier to manage and allocate across heterogeneous GPUs. This segmentation reduces the complexity of determining job allocation by creating consistent, interchangeable task units.
Solution Approach 2:
The microservice architecture creates universal task units that can be executed on any suitable GPU regardless of heterogeneity. This universality simplifies allocation logic by allowing the same scheduling mechanisms to handle diverse GPU types without requiring job-specific allocation rules.
3Productivity
If data parallelization-based training is performed in a heterogeneous cluster, then training capacity increases, but performance is reduced by straggler issues
Solution Approach 1:
The dynamic scheduling system monitors the execution status of microservices and can reassign straggler tasks to available GPUs. When some GPUs finish their tasks earlier than others, the scheduler dynamically redistributes the remaining microservices to maintain balanced execution and eliminate straggler effects.
Solution Approach 2:
The system implements feedback mechanisms where the scheduler continuously monitors GPU performance and job progress. Based on this feedback, it can identify straggling microservices and reassign them to different GPUs, thereby maintaining training performance while utilizing heterogeneous cluster capacity.
4Reliability
If a monolithic deep neural network model is trained, then the model can be trained as a unified structure, but it is difficult to utilize microservice systems for distributed processing
Solution Approach 1:
The patent segments the monolithic training job into microservices while maintaining the logical integrity of the neural network model. Each microservice processes a portion of the training data or a specific computational task, but the collective execution of all microservices preserves the unified model training process.
Solution Approach 2:
The job proxy and scheduler act as intermediaries that coordinate microservices to execute the monolithic model training. These intermediary components ensure that distributed microservices collectively perform the training task as if it were a unified process, maintaining model integrity while enabling distributed processing.
Data Source
AI summary
Disclosed is a training system performing training on a plurality of neural network models in parallel. The training system includes a first job proxy that receives a training request for a first neural network model and partitions a first training job corresponding to the first neural network model into first microservices, a second job proxy that receives a training request for a second neural network model and partitions a second training job corresponding to the second neural network model into second microservices, a scheduler that dynamically schedules the first microservices and the second microservices so as to be executed by heterogeneous processing units, a cluster that includes the heterogeneous processing units, sequentially executes the first microservices and sequentially executes the second microservices, and a distributed in-memory database that stores parameters generated in response to the execution of the first microservices and the second micro services.