Hierarchical Neural Architecture Search for ML Accelerator Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search (NAS) methods, such as Multi-trial and One-Shot NAS, are inefficient and resource-intensive, particularly on machine learning accelerators, due to long training times, memory constraints, and bias in search algorithms.
Innovation Solution
A hybrid and hierarchical NAS approach that partitions the search space into sub-search spaces, using Multi-trial NAS for initial exploration and One-Shot NAS within each sub-space, with early stopping criteria to optimize resource use and reduce bias.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If One-Shot NAS is used to explore the search space in a single search, then total search cost is reduced significantly, but bias is introduced that causes issues on finding the optimal candidate
Solution Approach 1:
The search space is divided into multiple sub-search spaces, and multiple One-Shot NAS searches are performed on different sub-spaces. This segmentation reduces the bias in each individual search while maintaining computational efficiency, allowing the system to find optimal candidates without the excessive cost of exhaustive search.
2Adaptability or versatility
If the supernetwork combines all model candidates in one shot, then weight sharing is achieved, but the large model size causes out of memory occurrence on ML accelerators
Solution Approach 1:
The supernetwork is divided into multiple sub-supernetworks, each corresponding to a sub-search space. This segmentation reduces the model size of each individual supernetwork, preventing out-of-memory errors on ML accelerators while maintaining weight sharing benefits within each sub-supernetwork.
Solution Approach 2:
The search space is partitioned along the dimension of model architecture parameters, creating multiple sub-search spaces. This dimensional partitioning allows the system to manage large search spaces without requiring a single large supernetwork, thus avoiding memory overflow while exploring diverse architecture options.
3Reliability
If Multi-trial NAS is used to explore the design space with separate trials, then stable results are achieved with no interference among trials, but each trial takes a long time making the search process prohibitively expensive
Solution Approach 1:
The search space is segmented into multiple sub-search spaces, and One-Shot NAS is applied to each sub-space. This approach maintains result stability by thoroughly exploring each sub-space while reducing the total search time compared to Multi-trial NAS, as weight sharing within sub-supernetworks accelerates the search process.
4Manufacturing precision
If the search space is large and requires 1000 or more samples to converge, then comprehensive exploration is achieved, but the search process becomes too expensive with 5 days training per trial
Solution Approach 1:
The large search space requiring 1000+ samples is divided into multiple smaller sub-search spaces. Each sub-space requires fewer samples to converge (e.g., 10-100 samples), reducing the training cost per trial from 5 days to a fraction of that time, while the collective exploration of all sub-spaces achieves comprehensive coverage of the original search space.
Data Source
AI summary
According to various implementations, generally disclosed herein is a hybrid and hierarchical neural architecture search (NAS) approach. The approach includes performing a search space partitioning scheme to divide the search space into sub-search spaces. The approach further includes performing a first type of NAS, such as a Multi-trial NAS, to cover a search across the sub-search spaces. The approach also includes performing a second type of NAS, such as a One-Shot NAS, to cover each sub-search space. The approach further includes automatically stopping the second type of NAS based on one or more early stopping criteria.