One-Shot Neural Architecture Search With Unlabeled Output Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search (NAS) techniques face challenges with limited labeled data, leading to reduced training quality, biased performance estimation, and increased variance, especially when using training data for both model training and architecture search, or relying on unlabeled proxy tasks with low correlation to the actual task.
Innovation Solution
Utilize the full labeled training data for model training and reserve it for supermodel development, employing unlabeled data for architecture search by comparing subnetwork outputs to the largest submodel within the supermodel, maintaining data independence and reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data is used for both model training and architecture search, then the quality of trained models is improved, but the performance estimation becomes biased and overfitting risk increases
Solution Approach 1:
The patent segments the data usage into two distinct phases: training phase where training data is used to train the supermodel, and search phase where the same training data is reused to evaluate subnetwork architectures. This segmentation allows the training data to serve dual purposes while maintaining clear phase boundaries that prevent overfitting during the search phase.
Solution Approach 2:
The supermodel is preliminarily trained on the training data before the architecture search begins. This preliminary training establishes a foundation that enables subsequent subnetwork evaluation using the same data without requiring additional labeled data, thereby resolving the contradiction between training quality and unbiased estimation.
2Measurement precision
If training data is split into separate parts for training and search, then unbiased performance estimation is achieved, but the quality of trained models deteriorates due to reduced training data
Solution Approach 1:
The training data is given multi-functionality by serving both as training data for the supermodel and as search data for evaluating subnetwork architectures. This universal usage of the training data eliminates the need to split the data, thereby maintaining training quality while enabling unbiased performance estimation through the two-stage procedure.
3Quantity of substance
If unlabeled proxy tasks are used for architecture search, then labeled data requirements are reduced, but the quality of found architectures deteriorates due to low correlation with actual task
Solution Approach 1:
Instead of using unlabeled proxy tasks, the patent copies the training data to serve as search data for evaluating subnetwork architectures. This copying approach ensures that the search process operates on the same data distribution as the actual task, maintaining high correlation and architecture quality while reducing labeled data requirements through the one-shot framework.
Data Source
AI summary
A computer-implemented method of neural architecture search. The method includes: training a supermodel by sampling an architecture and training the supermodel with the sampled architecture on labeled training data and updating weights of the supermodel by gradients with respect to the sampled models; determining Pareto-optimal submodels of the supermodel based on at least two performance metrics by iteratively carrying out the following steps: computing outputs of a reference model for the unlabeled data, wherein the reference model a largest submodel of the supermodel; sampling a plurality of submodels from the supermodel; computing by the submodels their outputs of the unlabeled data; computing a difference between the outputs of the reference model and the submodels; employing an optimization algorithm to iteratively sample and evaluate submodels based on a plurality of objectives.