Neural Architecture Search Pipeline for Speed-Accuracy Trade-off
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for neural architecture search (NAS) are either time-consuming or result in sub-optimal network architectures, especially when dealing with large search spaces.
Innovation Solution
A pipeline that sequentially performs training-free NAS, SuperNet/gradient-based search, and sampling method search to reduce the number of architectures that need to be trained, while ensuring high-quality architectures are obtained in minimal time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If training-free NAS is used to predict network performance without training, then the evaluation speed is improved, but the accuracy of performance prediction deteriorates
Solution Approach 1:
The evaluation process is segmented into multiple stages: training-free evaluation for rapid initial filtering, followed by training-based evaluation for accurate assessment of selected candidates. This segmentation allows different evaluation methods to be applied at appropriate stages, balancing speed and accuracy.
Solution Approach 2:
Training-free evaluation is performed as a preliminary action before actual training to quickly identify promising architecture candidates. This preliminary filtering reduces the number of architectures that require time-consuming training, while still maintaining the ability to identify high-performing models.
2Extent of automation
If SuperNet-based gradient search is used to search for optimal architecture, then the automation extent is improved, but the time consumption increases
Solution Approach 1:
The architecture search process is segmented into automated SuperNet-based search for exploration, followed by targeted training and evaluation for exploitation. This segmentation allows automation to handle the broad search space efficiently while concentrating computational resources on promising candidates.
Solution Approach 2:
Instead of performing exhaustive training on all candidate architectures, the system performs partial training or evaluation only on a selected subset of architectures identified by the automated search. This partial action significantly reduces time consumption while maintaining automation benefits.
3Manufacturing precision
If sampling method with many trainings is used to find optimal architecture, then the manufacturing precision is improved, but the productivity deteriorates
Solution Approach 1:
Training-free evaluation and automated search are performed as preliminary actions to identify high-potential architecture candidates before committing to time-consuming training processes. This ensures that full training is applied only to architectures with the highest likelihood of optimality.
Solution Approach 2:
The system performs training on a limited subset of architectures rather than exhaustively training all candidates. By combining automated search with selective training, the system achieves high optimization quality for the final selected architecture without the productivity loss of training all possible candidates.
Data Source
AI summary
Provided are a method, system, and device for a neural architecture search (NAS) pipeline for performing an optimized neural architecture search (NAS). The method may include obtaining a first search space comprising a plurality of candidate layers for a neural network architecture; performing a training-free NAS in the first search space to obtain a first set of architectures; obtaining a second search space based on the first set of architectures; performing a gradient-based search in the second search space to obtain a second set of architectures; performing a sampling method search utilizing the second set of architectures as an initial sample; and obtaining an output architecture as an output of the sampling method search.