Distributed Learning Server Dynamic Data Subset Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed learning in computer clusters faces delays due to differences in node computing performance and network speed, which can hinder the overall learning process.
Innovation Solution
A method and system for distributed learning that involves building a computer cluster by selecting worker nodes, splitting a training dataset into subsets, training AI models on each node, updating weights based on training results, and adjusting the number of data in each subset based on node operation times to synchronize training speeds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If distributed learning is performed using a computer cluster with multiple nodes, then the computing power and training capacity are improved, but delays occur due to differences in node computing performance and network speed
Solution Approach 1:
The patent implements dynamic data subset allocation where the master node adjusts the number of training data samples assigned to each worker node based on their actual training speeds. Fast nodes receive larger data subsets while slow nodes receive smaller subsets, creating a dynamic balancing mechanism that prevents fast nodes from idle waiting and slow nodes from becoming bottlenecks, thereby resolving the time loss caused by performance differences while maintaining high computing power utilization
Solution Approach 2:
The system changes the parameter of data subset size allocated to each worker node based on their training performance. By monitoring training speeds and adjusting the number of training data samples assigned to each node, the system optimizes the workload distribution parameter to achieve synchronized completion of training tasks across all nodes, eliminating the time delay caused by heterogeneous node performance
2Ease of manufacture
If data subsets are evenly distributed to all nodes, then the data distribution is simple and uniform, but nodes with different computing speeds cannot complete training simultaneously
Solution Approach 1:
The patent applies local quality by assigning different data subset sizes to different worker nodes based on their individual training capabilities. Instead of uniform distribution, each node receives a customized data allocation proportional to its computing speed, allowing fast nodes to process more data and slow nodes to process less, thereby achieving simultaneous completion and maximizing overall training efficiency while maintaining manageable distribution complexity
3Reliability
If the master node waits for all nodes to complete training, then all training results are collected, but the overall training time is extended by the slowest node
Solution Approach 1:
The master node dynamically adjusts data subset allocations in subsequent training rounds based on observed node performance. Nodes that complete training faster are assigned larger data subsets in the next round, while slower nodes receive smaller subsets. This dynamic adaptation ensures that the master node can proceed with weight updates as soon as sufficient training results are available, rather than waiting for all nodes, thereby reducing overall training time while maintaining result completeness
Solution Approach 2:
The system implements a feedback mechanism where the master node monitors the training completion status and performance of each worker node. Based on this feedback, the master node adjusts the data subset allocation for subsequent training iterations, assigning more data to fast nodes and less to slow nodes. This feedback-driven dynamic allocation allows the system to minimize waiting time while ensuring all necessary training results are eventually collected for reliable model updates
Data Source
AI summary
Provided is a method, performed by a server, of performing distributed learning. The server builds a computer cluster by selecting worker nodes that are to perform distributed learning, from among a plurality of nodes, wherein nodes in the computer cluster include the server that is a master node and the worker nodes. The server identifies, with respect to each of the nodes in the computer cluster, an operation time taken for each of the nodes in the computer cluster to perform training, and adjusts a number of data included in each of data subsets, based on the operation time of each of the nodes in the computer cluster, the data subsets being used in training of the nodes in the computer cluster.