Cross-Cluster Model Training Data Generation and Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cross-cluster model training methods are inefficient due to high communication bandwidth requirements, leading to increased training time, especially in data parallel and pipeline parallel modes, where large amounts of data need to be transmitted between clusters for model updates.
Innovation Solution
The proposed solution involves separating the tasks of generating training data and training a pre-trained model across different clusters, where the first cluster generates training data and sends it to the second cluster for model training, reducing the need for transmitting model parameters and utilizing lower-bandwidth communication between clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If cross-cluster model training is performed with data parallel or pipeline parallel mode, then model training capability is improved, but communication bandwidth requirement increases
Solution Approach 1:
The patent divides the model training system into multiple independent clusters, each capable of autonomous training. Within each cluster, multiple processing units work in parallel on different data batches or model components, eliminating the need for frequent inter-cluster parameter synchronization. This segmentation allows high-productivity parallel training while reducing inter-cluster communication bandwidth requirements.
2Manufacturing precision
If large amounts of data are transmitted between clusters for model updates, then model training accuracy is improved, but training time increases
Solution Approach 1:
The patent performs preliminary data processing and feature extraction within each cluster before model training. Sample data is pre-processed locally to generate optimized training inputs, reducing the amount of data that needs to be transmitted between clusters during the actual training process. This preliminary action maintains training accuracy while significantly reducing inter-cluster communication time.
Data Source
AI summary
A model training system includes at least one first cluster and a second cluster communicating with the at least first cluster. The at least one first cluster is configured to acquire a sample data set, generate training data according to the sample data set, and send the training data to the second cluster; and the second cluster is configured to train a pre-trained model according to the training data sent by the at least one first cluster.