Distributed Training Model Slicing and Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed training technologies in AI face challenges such as low automation, high single-point failure rates, and inefficient resource utilization, particularly in large-scale model training across various fields like computer vision and natural language processing.
Innovation Solution
The proposed solution involves a distributed training method based on end-to-end adaptive techniques, which includes slicing models into distributed operators and tensors, parsing computing resources to determine their attributes, and dynamically determining distribution strategies to optimize training efficiency and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional distributed training technology is used, then model training can be performed on large-scale data, but automation is low and manual intervention is required
Solution Approach 1:
The system performs self-detection of computing resource attributes, automatic model slicing, and automated distribution strategy determination without requiring manual intervention. The training framework automatically adapts to the computing resources available, eliminating the need for manual configuration and reducing operational complexity.
Solution Approach 2:
The system dynamically determines distribution strategies based on detected computing resource attributes. By changing the parameters of model slicing and distribution according to the detected resource characteristics, the system achieves high automation while adapting to different hardware configurations.
2Productivity
If distributed training is implemented across multiple computing resources, then training efficiency can be improved, but the single point failure rate increases
Solution Approach 1:
The model is sliced into multiple independent segments that can be distributed across different computing resources. This segmentation allows the training to continue on remaining segments if one computing resource fails, reducing the single point failure rate while maintaining training efficiency through parallel processing.
Solution Approach 2:
The system detects computing resource attributes beforehand and determines optimal distribution strategies that account for potential failures. By pre-configuring the distribution of model segments across resources based on their attributes, the system creates a resilient architecture that can withstand resource failures.
3Productivity
If computing resources are allocated for model training, then training can be performed, but resource utilization rate is inefficient
Solution Approach 1:
The system detects the attributes of available computing resources and uses this feedback information to automatically determine the optimal distribution strategy. This closed-loop approach ensures that computing resources are efficiently utilized by matching model segments to resources based on their actual capabilities, improving resource utilization rate.
Solution Approach 2:
The distribution strategy is dynamically determined based on the detected computing resource attributes rather than being fixed. This dynamic adaptation allows the system to optimize resource utilization by assigning model segments to the most suitable resources, improving overall productivity without requiring complex manual allocation.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
A distributed training method based on end-to-end adaption, a device and a storage medium. The method includes: obtaining (S101) slicing results by slicing a model to be trained; obtaining (S102) an attribute of computing resources allocated to the model for training by parsing the computing resources, in which the computing resources are determined based on a computing resource requirement of the model, computing resources occupied by another model being trained, and idle computing resources, and the attribute of the computing resources is configured to represent at least one of a topology relation and a task processing capability of the computing resources; determining (S103) a distribution strategy of each of the slicing results in the computing resources based on the attributes of the computing resources; and performing (S104) distributed training on the model using the computing resources based on the distribution strategy.