Distributed ML Training with Cascaded Node Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model training in distributed systems faces increased delay and reduced efficiency due to poor communication link quality of a single server node, leading to performance degradation.
Innovation Solution
A method for training a machine learning model in a distributed system where submodels are sequentially cascaded within node groups, allowing for intra-group information exchange without centralized communication, enabling flexible model parameter sharing and expanding datasets across node groups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If centralized training mode with a single server node is used, then model training can be performed in a simplified manner, but the overall model training delay increases due to deep channel fading of the single server node
Solution Approach 1:
The patent divides the centralized training system into multiple node groups, where each node group contains multiple nodes that can independently perform training operations. This segmentation eliminates the single point of failure and deep channel fading issues associated with a single server node, thereby reducing model training delay while maintaining system simplicity.
2Productivity
If splitting learning is used to reduce computing load of a single node, then communication overheads are reduced and sample size is expanded, but the overall AI model training delay increases when communication link quality is poor
Solution Approach 1:
The patent combines multiple nodes within node groups to perform collaborative training, where nodes can aggregate their computing resources and data. This merging approach allows the system to maintain the benefits of distributed computing while improving communication reliability through redundancy, thereby reducing training delay under poor communication conditions.
3Device complexity
If centralized training with a single server is used, then system structure is simple, but learning performance degrades due to deep channel fading
Solution Approach 1:
The patent assigns different training tasks and data subsets to different node groups, allowing each group to specialize in specific aspects of model training. This local quality differentiation improves learning performance by distributing the computational workload across multiple reliable nodes, avoiding the deep channel fading issues that affect centralized single-server systems.
Data Source
AI summary
This application discloses a method for training a machine learning model in a distributed system and a related apparatus. In the distributed system, an ith node in a node group obtains second data based on first data and a submodel in the ith node, where the first data is local data of the ith node or output data of an (i−1)th node in the same node group; performs gradient backpropagation based on third data, to obtain first gradient information of the ith node, where the third data is output data of an (i+1)th node in the same node group or local output data obtained based on the second data; receives a model parameter from at least one first node, where the first node is a node in a second node group; and updates a parameter of a local submodel based on the model parameter.