Parallel Neural Network Training via Odd-Even Layer Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks with multiple layers becomes increasingly time-consuming as the number of layers increases, necessitating a method to accelerate the forward and backward propagation processes.
Innovation Solution
The neural network is restructured into odd and even neural networks, allowing for parallel execution of forward and backward propagation by separating the layers, with error gradients computed and propagated independently through each set to update parameters efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of layers in the neural network model is increased, then the model's processing capability and accuracy are improved, but the training time increases significantly
Solution Approach 1:
The patent divides the neural network into separate odd-layer model and even-layer model, allowing independent parallel training of each subset. This segmentation enables simultaneous processing of multiple layers without sequential dependencies, thereby reducing overall training time while maintaining the full depth of the network for accuracy.
Solution Approach 2:
The patent introduces a new dimension of parallelism by separating layers based on their odd/even indexing. Instead of training layers sequentially in a single timeline, the network structure is transformed into multiple parallel training dimensions, where odd and even layers can be processed concurrently, effectively adding a parallel processing dimension to the training architecture.
2Reliability
If forward and backward propagation are performed sequentially through each layer, then parameter updates are accurate, but the total computation time increases
Solution Approach 1:
The propagation process is segmented into separate odd-layer propagation and even-layer propagation pathways. Each pathway can execute independently and in parallel, maintaining the mathematical correctness of gradient computation while eliminating sequential bottlenecks between alternating layers.
Solution Approach 2:
The patent merges the training of odd and even layers into a unified parallel framework where both pathways operate simultaneously. By combining the results from both parallel propagations, the system achieves both high accuracy and improved training efficiency.
Data Source
AI summary
A neural network structure is separated into an odd neural network including only the odd layers and an even neural network including only the even layers. In order to allow for parallel execution, for forward propagation a second input is generated from the original input, while for backward propagation a second error gradient is generated. Parallel execution may accelerate the forward and backward propagation operations without significant change in accuracy of the model. Additionally, restructuring a single neural network into two or more parallel neural networks may reduce the total time needed for training.