Distributed Deep Neural Network Training via Subnetwork Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training of deep neural networks becomes time-consuming due to the explosive growth of network parameters and training samples, and existing parallel training methods face challenges in reducing network delay in distributed training setups.
Innovation Solution
A method and apparatus that divide a deep neural network into subnetworks and training samples into subsets, utilizing a distributed cluster architecture and a preset scheduling method to minimize network delay through data localization, allowing simultaneous training and synchronization of subnetworks to accelerate the training process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed parallel training is used to reduce training time, then productivity is improved, but network delay increases causing loss of time
Solution Approach 1:
The patent divides the deep neural network into multiple subnetworks and training samples into multiple subsets, enabling parallel training across multiple processors. This segmentation allows the system to achieve faster overall training productivity while managing network communication overhead through localized data processing.
Solution Approach 2:
The patent implements data localization by storing data subsets locally at each processor rather than requiring centralized data access. This local quality approach minimizes network delay by reducing the frequency and volume of data transmissions between processors, directly addressing the time loss from network communication while maintaining parallel training productivity.
2Productivity
If data is transmitted between multiple processors for distributed training, then model training capability is improved, but data transmission time increases
Solution Approach 1:
The patent performs preliminary data distribution by dividing the training dataset into multiple subsets and assigning them to different processors before training begins. This preliminary action reduces subsequent data transmission time by ensuring each processor has the necessary data locally available, while still enabling distributed training capability across the network.
Solution Approach 2:
Each processor is assigned specific data subsets to process locally, reducing the need for frequent inter-processor data transmissions. This local quality approach maintains distributed training capability while minimizing the time lost to data transmission by keeping data processing localized to each processor's memory and storage.
3Productivity
If multiple processors are used for parallel training, then training efficiency is improved, but synchronization complexity increases
Solution Approach 1:
The patent implements a synchronization mechanism that collects training progress information from multiple processors and coordinates their operation. This feedback-based approach maintains high training efficiency by keeping processors working in parallel while managing synchronization complexity through systematic information collection and coordination protocols.
Solution Approach 2:
The patent divides the training task into independent subnetworks that can be trained in parallel with reduced synchronization requirements. This segmentation of the overall training task into smaller, more manageable units maintains high training efficiency while reducing synchronization complexity compared to coordinating a single large model across all processors.
Data Source
AI summary
Embodiments of the present invention provide a method and apparatus for accelerating distributed training of a deep neural network. The method comprises: based on parallel training, the training of deep neural network is designed as a distributed training mode. A deep neural network to be trained is divided into multiple sub-networks. A set of training samples is divided into multiple subsets of samples. The training of the deep neural network to be trained is performed with the multiple subsets of samples based on a distributed cluster architecture and a preset scheduling method. The multiple sub-networks are simultaneously trained so as to fulfill the distributed training of the deep neural network. The utilization of the distributed cluster architecture and the preset scheduling method may reduce, through data localization, the effect of network delay on the sub-networks under distributed training; adapt the training strategy in real time; and synchronize the sub-networks trained in parallel. As such, the time required for the distributed training of the deep neural network may be reduced and the training efficiency of the deep neural network may be improved.