Microservice Training System for Dynamic Resource Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In shared cluster environments, training deep neural networks (DNNs) is challenging due to limited resources and the difficulty in efficiently managing and scheduling resources using the microservice execution method, which is typically suited for serverless systems but not optimized for long-running tasks like DNN training.
Innovation Solution
A training system that partitions a DNN training job into microservices and schedules them across multiple logical workers, dynamically allocating processing units based on availability, allowing for flexible resource utilization and efficient scheduling, even with fewer resources than required, by using a job proxy and scheduler to manage and allocate resources effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a shared cluster environment is used to execute DNN training jobs, then resource utilization is improved, but resource allocation efficiency deteriorates due to limited resources and difficulty in managing long-running tasks
Solution Approach 1:
The training job is divided into multiple microservices, each representing a independent processing unit that can be scheduled separately. This segmentation allows the scheduler to allocate limited processing units more flexibly across multiple users and jobs, improving resource allocation efficiency while maintaining high resource utilization in the shared cluster environment
Solution Approach 2:
The system dynamically schedules microservices to processing units based on real-time availability status. When processing units become available, pending microservices are automatically scheduled and executed. This dynamic approach allows the system to adapt to changing resource conditions, improving both resource utilization and allocation efficiency
2Adaptability or versatility
If microservice execution method is used for DNN training, then resource flexibility is improved, but scheduling complexity increases due to the need to manage multiple logical workers and their microservices
Solution Approach 1:
The scheduler acts as an intermediary between logical workers and processing units. It receives scheduling requests from logical workers, determines availability of processing units, and allocates resources accordingly. This intermediary layer simplifies the overall scheduling complexity by centralizing the decision-making process while maintaining resource flexibility
Solution Approach 2:
The scheduler is designed to handle multiple types of scheduling scenarios universally - it can schedule microservices from different logical workers, handle various availability statuses, and manage both sequential and parallel execution. This universal scheduling mechanism reduces complexity by providing a single, unified approach to resource management
3Adaptability or versatility
If the number of available processing units is less than the number of logical workers, then resource utilization is improved, but training performance deteriorates due to sequential scheduling requirements
Solution Approach 1:
By segmenting the training job into microservices, the system can efficiently pack multiple microservices into available processing units. Even with fewer processing units than logical workers, the segmented microservices can be scheduled to maximize utilization of available resources while maintaining acceptable training performance through phased execution
Data Source
AI summary
Provided are a training system and an operating method thereof. The training system includes a job proxy configured to partition a training job corresponding to a neural network model into a plurality of microservices respectively executed by a plurality of logical workers, and a scheduler configured to schedule the plurality of microservices for a plurality of processing units, respectively, wherein the plurality of microservices includes a plurality of first microservices executed by a first logical worker among the plurality of logical workers and a plurality of second microservices executed by a second logical worker among the plurality of logical workers, and the scheduler is configured to schedule the plurality of first microservices and the plurality of second microservices to any one processing unit among the plurality of processing units based on an availability status of the plurality of processing units.