Distributed Neural Network Training Under Changing GPU Resources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently performing distributed learning on neural network models due to heterogeneous GPUs with different performance and changes in resource availability, leading to inefficient resource utilization.
Innovation Solution
An electronic device and method that dynamically adjusts parallelism processes (pipeline, data, and tensor parallelism) based on resource changes, optimizing computation by reallocating layers and resources to maximize data parallelism and minimize memory usage, and selecting the optimal parallelism method based on execution time and memory constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed learning is performed using heterogeneous GPUs with different performance, then resource utilization is improved, but computation efficiency deteriorates
Solution Approach 1:
The patent implements dynamic parallelism method selection by monitoring resource availability in real-time. When resources change (e.g., GPU failures, availability changes), the system dynamically switches between different parallelism strategies (data parallelism, pipeline parallelism, tensor parallelism) to adapt to the current resource configuration, thereby maintaining computation efficiency despite heterogeneous hardware conditions
Solution Approach 2:
The system changes operational parameters by selecting different parallelism methods based on resource conditions. When certain GPUs become unavailable or performance characteristics change, the system modifies the parallelism strategy parameters (e.g., switching from data parallelism to pipeline parallelism) to optimize computation efficiency for the remaining resources
2Adaptability or versatility
If the number of resources is changed during distributed learning, then resource flexibility is improved, but learning stability deteriorates
Solution Approach 1:
The system implements feedback mechanisms to monitor resource availability and learning progress. When resource changes are detected, the system uses feedback information to determine the optimal parallelism method and adjusts the training process accordingly, maintaining stability by ensuring continuous progress despite resource fluctuations
Solution Approach 2:
The patent performs preliminary actions by pre-defining multiple parallelism methods and their corresponding resource requirements before resource changes occur. This allows the system to quickly switch between predefined strategies when resource changes are detected, minimizing disruption to learning stability
3Adaptability or versatility
If parallelism process is performed multiple times to adapt to resource changes, then adaptability is improved, but computation time increases
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing multiple parallelism methods and their performance characteristics before resource changes occur. This allows the system to quickly select and switch between predefined strategies when resource changes are detected, avoiding the need to perform complete parallelism processes from scratch and thereby reducing time loss
Solution Approach 2:
The patent implements dynamic selection of parallelism methods based on real-time resource monitoring. Rather than performing all possible parallelism processes, the system dynamically chooses the most appropriate method for current resource conditions, reducing unnecessary computation time while maintaining adaptability
Data Source
AI summary
An electronic device and a controlling method thereof are provided. The electronic device includes memory, comprising one or more storage media, storing instructions and configured to store information on a neural network model and information on a plurality of resources for performing distributed learning on the neural network model, and a processor communicatively coupled to the memory and configured to perform a parallelism process including pipeline parallelism, data parallelism, and tensor parallelism based on the information on the neural network model and the information on the plurality of resources, wherein the instructions, when executed by the processor, cause the electronic device to acquire a first computation amount when performing the distributed learning from a time when a change in the plurality of resources is detected to a next checkpoint using the plurality of resources before the change, if the change is detected while performing the distributed learning according to a result of performing the parallelism process, perform the parallelism process again based on the information on the plurality of changed resources, acquire a second computation amount when performing the distributed learning from the time when the change is detected to the next checkpoint using the plurality of changed resources, as the result of the parallelism performed again, and perform the distributed learning by a method corresponding to a smaller computation amount of the first computation amount and the second computation amount.