Distributed Training Information Fusion via Key Node Selection and Parameter Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning model training methods face challenges due to complex algorithmic logic and intensive computation, which limits performance and requires large datasets and models. Additionally, inefficient communication methods make it difficult to apply these methods in time-sensitive scenarios, as they can take weeks to train neural network models.
Innovation Solution
A method for information fusion in a distributed training system, where local parameters from workers are fused to obtain a global parameter. This involves selecting key nodes based on parameter deviations, fusing their local parameters, and updating the global parameter, which is then sent back to the workers for the next round of training. The method also includes a compression algorithm to reduce communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all workers participate in information fusion, then model training accuracy is improved, but communication overhead and training time increase significantly
Solution Approach 1:
The patent segments the worker population into key nodes and ordinary nodes. Only key nodes participate in information fusion operations, while ordinary nodes perform local training independently. This segmentation reduces the number of participants in communication-intensive operations from all workers to a selected subset, thereby reducing communication overhead and training time while maintaining model accuracy through careful selection of key nodes based on parameter deviation metrics
2Productivity
If compression algorithms are applied to reduce communication traffic, then communication efficiency is improved, but parameter precision may be degraded
Solution Approach 1:
The patent applies compression algorithms selectively and partially - compression is applied to parameter transmissions during information fusion operations, but the system maintains full precision for critical computations. The compression level and method are chosen to provide sufficient reduction in communication traffic while preserving the essential information needed for accurate model convergence, balancing communication efficiency with parameter precision
3Reliability
If deep learning models use large datasets and large models, then model performance is improved, but computational complexity and resource requirements increase
Solution Approach 1:
The patent segments the computational workload across multiple workers in a distributed system, with each worker handling a portion of the training tasks. Information fusion is performed only among key nodes rather than all workers, reducing the computational complexity of the aggregation step. This segmentation allows the system to handle large datasets and models by distributing computation while reducing the complexity burden on individual nodes and the overall system
4Speed
If communication frequency between workers and server is increased, then model convergence speed is improved, but communication overhead and system load increase
Solution Approach 1:
The patent implements periodic information fusion operations triggered by communication triggering conditions rather than continuous communication. The system uses asynchronous communication mechanisms where workers can continue local training independently, and information fusion is performed periodically when key nodes are selected and communication conditions are met. This periodic action maintains model convergence speed by ensuring regular updates while significantly reducing communication overhead compared to synchronous continuous communication
Data Source
AI summary
The embodiments of the present disclosure relate to the technical field of computers. Disclosed are a method and apparatus for information fusion, a method and apparatus for data communication, and an electronic device and a non-transitory computer-readable storage medium. The method for information fusion includes: in response to that a communication triggering condition is met, acquiring a local parameter of each of workers in a distributed training system, where the communication triggering condition is that all key nodes complete tasks of the current round of training; selecting N key nodes participating in the next round of training, and fusing local parameters of the N key nodes to obtain a global parameter; and sending the global parameter to each of the workers, and sending a training command to the key nodes to the key nodes execute tasks of next round of training based on the global parameter.