Data-Parallel Model Training With Weighted Loss Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large models with deep learning is extremely costly due to the need for significant hardware investment and daily maintenance, and existing parallel algorithms are inefficient and complex, making it infeasible to train models on a single computing device.
Innovation Solution
A method for training models using data parallelism that involves obtaining training losses from multiple terminals, calculating a weighted average of these losses, and updating local models based on the weighted loss, with simplified synchronization and aggregation processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional parallel algorithms are used to train large models, then model training becomes feasible on distributed systems, but the system complexity and cost increase significantly
Solution Approach 1:
The patent segments the model training process by distributing data across multiple terminals rather than splitting the model itself. Each terminal holds a copy of the model and processes different data partitions independently, then synchronizes results. This approach makes large model training feasible on distributed systems while avoiding the complexity of model parallelism.
Solution Approach 2:
The patent uses copying by having each terminal maintain a copy of the model parameters. Instead of sharing a single model across terminals (which would require complex synchronization), each terminal works with its own model copy, processes local data, and then updates the shared parameters. This simplifies the system architecture while enabling distributed training.
2Productivity
If more hardware resources are allocated for training large models, then training capacity increases, but the cost and resource consumption increase proportionally
Solution Approach 1:
The patent applies partial action by having each terminal process only a subset of the data rather than requiring each terminal to handle the complete dataset. This allows the system to achieve high training capacity using multiple terminals with smaller individual resource requirements, rather than needing a single terminal with excessive computational power.
Solution Approach 2:
The patent merges the computational results from multiple terminals by aggregating their individual model updates. Each terminal processes its local data independently and contributes to a global model update, combining the strengths of multiple smaller computations to achieve the equivalent of a single large-scale computation with reduced individual resource consumption.
3Reliability
If traditional data parallelism is implemented with full parameter synchronization, then model accuracy is maintained, but communication overhead and training time increase
Solution Approach 1:
The patent applies preliminary action by performing local data processing and model updates before synchronization. Each terminal completes its local computations using its data partition, then only the aggregated results need to be communicated and synchronized. This preliminary local processing reduces the communication overhead and time required for full parameter synchronization.
Solution Approach 2:
The patent extracts only the necessary synchronization information from the full parameter set. Rather than synchronizing all model parameters simultaneously, the system extracts and synchronizes only the aggregated updates from each terminal's local computations, reducing communication overhead while maintaining model accuracy.
Data Source
AI summary
A method for training a model based on data parallelism and a terminal. The model comprises local models trained at training terminals, respectively, and the method comprises: obtaining, by a first terminal, respective training losses of the training terminals; and calculating, by the first terminal, a weighted average of the training losses to obtain a weighted training loss, wherein the weighted training loss is for updating a parameter of the local model trained at each of the training terminals.