Hybrid Data-Model Parallelism for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training techniques face inefficiencies with data parallelism in weight-heavy layers due to costly chip-to-chip operations and with model parallelism in feature-heavy layers due to extensive data replication and communication overhead.
Innovation Solution
Implementing a hybrid parallelism technique that dynamically switches between data and model parallelism based on the characteristics of each neural network layer, utilizing different communication directions to optimize bandwidth usage and minimize inefficiencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data parallelism is used in weight-heavy layers, then each processor can process complete batches independently, but costly chip-to-chip weight synchronization operations are required
Solution Approach 1:
The patent segments the weight data distribution across processors differently than the data batch distribution. In weight-heavy layers, weights are divided into segments assigned to different processors, allowing parallel weight updates without requiring full weight replication across all processors. This segmentation enables independent batch processing while eliminating costly synchronization operations.
Solution Approach 2:
The patent applies different parallelism strategies to different layers based on their characteristics. For weight-heavy layers, it uses a customized approach where processors maintain local weight segments rather than requiring global weight synchronization. This local quality approach optimizes each layer's processing based on its specific weight-to-feature ratio.
2Loss of time
If model parallelism is used in feature-heavy layers, then weight replication is avoided, but extensive input data replication and communication overhead occur
Solution Approach 1:
The patent dynamically selects between data parallelism and model parallelism for each layer based on the weight-to_feature ratio. This dynamic adaptation allows the system to use data parallelism for feature-heavy layers (avoiding input replication) and model parallelism for weight-heavy layers (avoiding weight replication), optimizing performance and energy efficiency for each specific layer type.
3Quantity of substance
If model parallelism is used, then weights are not replicated across processors, but all processors must receive and communicate all activations before the next layer can begin
Solution Approach 1:
The patent segments the output features and assigns them to different processors in a manner that allows independent processing. Instead of requiring all processors to wait for complete activation propagation, each processor can finalize its assigned feature computations and pass results forward, enabling overlapping execution and improved throughput while maintaining weight efficiency.
Data Source
AI summary
The embodiments herein describe hybrid parallelism techniques where a mix of data and model parallelism techniques are used to split the workload of a layer across an array of processors. When configuring the array, the bandwidth of the processors in one direction may be greater than the bandwidth in the other direction. Each layer is characterized according to whether they are more feature heavy or weight heavy. Depending on this characterization, the workload of an NN layer can be assigned to the array using a hybrid parallelism technique rather than using solely the data parallelism technique or solely the model parallelism technique. For example, if an NN layer is more weight heavy than feature heavy, data parallelism is used in the direction with the greater bandwidth (to minimize the negative impact of weight reduction) while model parallelism is used in the direction with the smaller bandwidth.