Neural Architecture Search Using Gradient-Based Sub-Space Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural architecture search (NAS) methods are either time-consuming and resource-intensive due to their iterative nature or constrained by memory limitations, making it difficult to find optimal neural network architectures efficiently.
Innovation Solution
Performing a plurality of gradient-based searches on sub-spaces to distribute hardware requirements and optimize initial seeds for a sampling method, allowing for an expanded search space and increased flexibility in neural network structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If gradient-based search is performed on a large search space, then manufacturing precision of neural network architecture is improved, but device complexity and resource consumption increase
Solution Approach 1:
The patent divides the large search space into multiple sub-spaces, each corresponding to different portions or aspects of the neural network architecture. Multiple gradient-based searches are performed in parallel on these sub-spaces, distributing the computational load across multiple hardware resources while collectively covering the entire search space, thereby achieving high architecture optimization without overwhelming single-device complexity
2Adaptability or versatility
If iterative sampling method is used for neural architecture search, then adaptability of architecture design is improved, but loss of time increases
Solution Approach 1:
The patent performs gradient-based searches on sub-spaces before conducting the final sampling method search. These preliminary gradient-based searches optimize initial seeds by identifying promising regions in the search space, so that when the sampling method subsequently explores the full search space, it starts from pre-optimized points rather than random samples, significantly reducing the time required to achieve high-quality architecture designs
Data Source
AI summary
Provided are method, system, and device for performing an optimized neural architecture search (NAS) by using both a gradient-based search and a sampling method on a search space. The method may include obtaining a first search space comprising a plurality of candidate layers for a neural network architecture; performing a gradient-based search in the first search space to obtain a first architecture; performing a sampling method search utilizing the first architecture as an initial sample; and obtaining a second architecture as an output of the sampling method search.