Neural Network Training via Super Network Parameter Inheritance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training neural networks are computationally burdensome and inefficient due to the need for extensive training from scratch, which hampers their performance and application in data processing tasks like classification and recognition.
Innovation Solution
The method involves training a super network with multiple candidate sub-structures in parallel, selecting a target sub-structure for each layer, and using the inherited network parameters from the super network as initial parameters for the sub-network, thereby reducing the computational burden and improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing methods are used to train a neural network from scratch, then the network can achieve required performance, but the computation burden is large and training efficiency is low
Solution Approach 1:
The patent applies preliminary action by pre-training a super network that contains multiple candidate sub-structures in parallel. The super network is trained beforehand to learn optimal parameters for all candidate sub-structures, so that when a specific sub-network is needed, the parameters are already available or can be quickly adapted, eliminating the need to train from scratch and significantly improving training efficiency while reducing computation burden
2Productivity
If a super network with multiple candidate sub-structures is trained, then training efficiency is improved through parameter inheritance, but the device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the super network into multiple independent candidate sub-structures, each with its own set of parameters. This segmentation allows the system to inherit and reuse parameters across different sub-structures, reducing the overall complexity compared to training each network separately while maintaining the ability to select the best-performing sub-structure for specific tasks
Data Source
AI summary
A method for training a neural network, includes: training a super network to obtain a network parameter of the super network, wherein each network layer of the super network includes multiple candidate network sub-structures in parallel; for each network layer of the super network, selecting, from the multiple candidate network sub-structures, a candidate network sub-structure to be a target network sub-structure; constructing a sub-network based on target network sub-structures each selected in a respective network layer of the super network; and training the sub-network, by taking the network parameter inherited from the super network as an initial parameter of the sub-network, to obtain a network parameter of the sub-network.