Distributed Neural Network Training Without Gradient Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed learning methods for training deep neural networks incur significant communication overhead due to the use of distributed stochastic gradient descent, which is inefficient and costly.
Innovation Solution
The proposed solution involves transforming data into higher-dimensional structures using discrete Fourier or cosine transforms, allowing sub-networks to be trained independently without communication, and combining their outputs through weighted sums.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed stochastic gradient descent is used to train deep neural networks, then network training can be performed in a distributed manner, but communication overhead increases significantly
Solution Approach 1:
The patent segments the training process by decomposing the original neural network into multiple independent sub-networks, each processing different transformed versions of the input data. This segmentation allows parallel training without requiring communication between nodes, as each sub-network is trained independently on its transformed data portion.
Solution Approach 2:
The patent applies dimensionality transformation by converting input data into higher-dimensional spaces using transform functions (e.g., Fourier transforms, wavelet transforms). This dimensionality change creates distinct transformed data representations that can be processed by different sub-networks independently, eliminating the need for parameter synchronization and reducing communication overhead.
2Reliability
If distributed stochastic gradient descent with central coordination is used, then parameter updates can be coordinated across nodes, but communication overhead becomes O(HL·S)
Solution Approach 1:
The patent segments both the data and the network architecture into independent components. Each node receives a segment of transformed data and trains a corresponding sub-network independently. This segmentation eliminates the need for centralized coordination of parameter updates, as each node operates autonomously on its segmented data portion.
Solution Approach 2:
Each distributed node performs self-service by independently transforming its local data and training its own sub-network without requiring parameter synchronization with other nodes. The transform function is applied locally at each node, enabling autonomous operation and eliminating communication overhead for parameter coordination.
3Productivity
If traditional distributed learning methods are used, then multiple nodes can collaborate on training, but the model size and computational complexity remain high
Solution Approach 1:
The patent segments the original large model into multiple smaller sub-networks, each with reduced parameter size. This segmentation allows multiple nodes to collaborate by each training a smaller sub-network independently, reducing the computational burden and model size requirements at each node while maintaining collaborative training capabilities.
Solution Approach 2:
By transforming data into higher-dimensional spaces, the patent enables the use of smaller, simpler sub-networks that can process the transformed data effectively. The dimensionality transformation captures essential features in the transformed space, allowing reduced model complexity while maintaining training effectiveness across distributed nodes.
Data Source
AI summary
Mechanisms for training a distributed neural network are provided, the mechanisms including: for each of a plurality of sub-networks: performing, using a hardware processor, a transform on data in a data structure having at least two dimensions to provide training data having a higher dimensionality than the at least two dimensions; and training the sub-network using the training data independently of other of the plurality of subnetworks. In some of these embodiments, the at least two dimensions is two dimensions. In some of these embodiments, the training data is stored in a three-dimensional structure. In some of these embodiments, the transform is a discrete Fourier transform. In some of these embodiments, the transform is a discrete cosine transform.