Distributed Sub-Model Training to Cut Bandwidth and Waiting Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training a complex machine learning model across multiple compute nodes is inefficient due to high bandwidth usage and blocking processes caused by transmitting training data between nodes, leading to increased waiting times.
Innovation Solution
A data-centric approach where sub-models are deployed on multiple compute nodes, with training data fixed, and sub-models transmitted between nodes, allowing for parallel processing of update parameters and reduced network bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If training data is transmitted between compute nodes during distributed training, then the machine learning model can be trained across multiple compute nodes, but the bandwidth is occupied heavily and compute nodes experience waiting delays
Solution Approach 1:
The patent inverts the conventional distributed training approach by having compute nodes transmit sub-models to a first compute node instead of transmitting training data to multiple nodes. The first compute node then determines update parameters and transmits them back to the respective compute nodes. This inversion eliminates the need for multiple nodes to wait for training data, thereby reducing waiting time and improving training efficiency.
Solution Approach 2:
The first compute node acts as an intermediary in the training process. It receives sub-models from multiple compute nodes, performs the computation to determine update parameters, and then distributes these parameters back to the respective nodes. This intermediary approach centralizes the data aggregation and parameter computation, allowing parallel processing and eliminating the blocking waiting time that occurs when nodes exchange training data directly.
2Productivity
If training data is transmitted between compute nodes, then distributed training can be performed, but a large amount of bandwidth is consumed
Solution Approach 1:
Instead of transmitting training data from the first compute node to multiple other nodes (which consumes large bandwidth), the patent inverts the flow by having multiple compute nodes transmit their sub-models to the first compute node. The update parameters generated are then transmitted back to the respective nodes. Since sub-models and update parameters are much smaller in size than the full training data set, the bandwidth consumption is significantly reduced.
3Adaptability or versatility
If the machine learning model structure becomes more complex to handle complicated tasks, then the model can perform better tasks, but training at a single compute node becomes difficult
Solution Approach 1:
The patent applies segmentation by dividing the complex machine learning model into multiple sub-models that can be distributed across different compute nodes. Each compute node maintains and trains its local sub-model independently using local training data. This segmentation allows the overall model to achieve high complexity and versatility while each individual node handles only a manageable portion, making distributed training feasible and efficient.
Data Source
AI summary
Provided are a method, an apparatus, a device, and a medium for training a machine learning model. The machine learning model includes a first sub-model and a second sub-model, the first sub-model is located at a first compute node in a computing system, and the second sub-model is located at a second compute node in the computing system. In the method, at the first compute node, a first set of training data for training the machine learning model is received. The second sub-model is obtained from the second compute node. The first set of training data is input into the first sub-model and the obtained second sub-model respectively to determine a first update parameter for updating the first sub-model and a second update parameter for updating the second sub-model. The second update parameter is transmitted to the second compute node.