Backbone Neural Architecture Search Through Partwise SuperNet Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large CNN models, known as SuperNet, is time-consuming due to their size and the gap between SuperNet and SubNet structures, which are inefficient and require significant human expertise.
Innovation Solution
A neural architecture searching apparatus and method that trains a supernetwork by replacing target layers with candidate layers, using fully-connected layers, and trains them part by part, followed by evaluation and selection of the best-performing parts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If a large SuperNet model with multiple candidate CNN layers is trained to enable Neural Architecture Search, then the search can automatically optimize CNN architecture, but the training time becomes extremely long
Solution Approach 1:
The patent segments the SuperNet training process into multiple stages: first training only the fully-connected layers while freezing CNN layers, then progressively training candidate CNN layers in groups. This segmentation allows the large model to be trained efficiently by breaking down the training workload into manageable parts, significantly reducing overall training time while maintaining automation.
Solution Approach 2:
The patent performs preliminary training of the fully-connected layers before training the CNN candidate layers. This preliminary action prepares the feature extraction and classification components in advance, creating a solid foundation that accelerates subsequent training of the CNN layers and reduces total training time.
2Adaptability or versatility
If multiple candidate CNN layers are trained simultaneously in parallel, then the architecture search comprehensiveness is improved, but the training complexity and time increase significantly
Solution Approach 1:
The patent segments the candidate CNN layers into multiple groups that are trained in sequence rather than all simultaneously. Each group is trained independently with controlled complexity, making the training process more manageable while still achieving comprehensive architecture search by evaluating multiple candidates across different groups.
Solution Approach 2:
The patent trains a subset of candidate CNN layers in each stage rather than all candidates at once. This partial action approach reduces training complexity at each step while maintaining comprehensiveness by progressively evaluating different candidate groups and selecting the best performing architectures.
3Adaptability or versatility
If the SuperNet structure maintains multiple parallel CNN layers throughout training, then the architecture search space is preserved, but the gap between SuperNet and final SubNet structure becomes large requiring extensive retraining
Solution Approach 1:
The patent segments the candidate CNN layers into groups that are progressively trained and evaluated. This segmentation allows the SuperNet structure to gradually transition toward the final SubNet configuration, reducing the structural gap and minimizing retraining time since each segment is already optimized before moving to the next stage.
Solution Approach 2:
The patent dynamically adjusts the training process by progressively enabling different groups of candidate CNN layers. This dynamic approach allows the network structure to evolve from the initial SuperNet configuration toward the final SubNet structure in a controlled manner, reducing the gap between intermediate and final architectures.
Data Source
AI summary
Technical ProblemTo provide a time efficient Neural Architecture Search for the Backbone block of Computer Vision task.Solution to ProblemA neural architecture searching apparatus comprises building means (11) to build a supernetwork, wherein a target layer of the supernetwork to be optimized is replaced by a plurality of candidate layers, and the supernetwork comprises a plurality of fully-connected layers; training means (12) to train the supernetwork, wherein the plurality of candidate layers are trained part by part, and the plurality of fully-connected layers are trained correspondingly to the part of the plurality of candidate layers; and selecting means (13) to evaluate the trained supernetwork and select a part of the plurality of candidate layers which corresponds to the best performing part of the plurality of fully-connected layers.