Asynchronous Neural Network Training via Distributed Subgraph Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network training is computationally expensive and resource-intensive, especially for architectures with varying control flow paths, making it difficult to exploit speed-ups from graphics processing units and requiring complex data flattening and padding, which hides underlying structure and reduces training effectiveness.
Innovation Solution
A distributed neural network training system with a network of worker nodes and a control node that implements parallelized message passing operations, allowing asynchronous updates of subgraph parameters, enabling efficient training without the need for graphics processing units and allowing online training for continuous updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If graphics processing units are used to speed up neural network training, then training speed is improved, but device complexity and resource requirements increase
Solution Approach 1:
The patent segments the neural network into multiple subgraphs distributed across different worker nodes. Each worker node independently processes a portion of the training data and performs local parameter updates, eliminating the need for centralized GPU processing while maintaining parallel training capabilities.
Solution Approach 2:
Each worker node autonomously performs forward propagation, backward propagation, and parameter updates for its assigned subgraph without requiring centralized coordination. The asynchronous update mechanism allows workers to independently adjust parameters based on local gradients, achieving self-organized parallel training.
2Loss of time
If training data instances are batched together, then training time is reduced, but data structure is lost and training effectiveness decreases
Solution Approach 1:
The patent segments both the neural network and training data, assigning specific data instances to specific worker nodes based on their associated subgraphs. This segmentation allows individual data instances to be processed without flattening or padding, preserving their original structure while enabling parallel processing.
Solution Approach 2:
Each worker node processes data with local specificity, maintaining the unique characteristics and structure of individual data instances. The local processing approach ensures that data-specific features are preserved while still achieving parallel training through distributed computation.
3Stability of the object's composition
If synchronous updates are used for neural network parameters, then training stability is improved, but training time increases due to waiting for all workers
Solution Approach 1:
The patent implements dynamic asynchronous updates where worker nodes can immediately apply parameter updates to their local subgraphs without waiting for other workers. This dynamic approach allows the system to adapt to varying computation speeds of different workers, eliminating idle waiting time while maintaining training progress through continuous updates.
Solution Approach 2:
The asynchronous update mechanism ensures continuous training progress by allowing each worker to perform useful computations and updates without being blocked by slower workers. The training process maintains continuous forward and backward propagation across all workers simultaneously, maximizing resource utilization and reducing overall training time.
Data Source
AI summary
A neural network training apparatus is described which has a network of worker nodes each having a memory storing a subgraph of a neural network to be trained. The apparatus has a control node connected to the network of worker nodes. The control node is configured to send training data instances into the network to trigger parallelized message passing operations which implement a training algorithm which trains the neural network. At least some of the message passing operations asynchronously update parameters of individual subgraphs of the neural network at the individual worker nodes.