Unbiased Neural Architecture Search via Dynamic Edge Probability Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural architecture search methods often introduce implicit preferences for certain paths in one-shot models, leading to biased exploration of the search space and potentially missing superior architectures.
Innovation Solution
The method initializes edge probabilities such that all paths from the input to the output node have equal probability, allowing for unbiased exploration of the search space and enabling the discovery of more superior architectures by randomly drawing paths and adjusting their probabilities during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional initialization methods are used where all outgoing edges of a node have the same probability, then the initialization is simple and uniform, but this introduces implicit preference for certain paths and biases the exploration of the search space
Solution Approach 1:
The patent changes the probability parameters of edges in the computational graph during the sampling process. Instead of using fixed uniform probabilities, the method dynamically adjusts edge probabilities based on training performance, allowing superior architectures to be identified and selected while maintaining initial simplicity in the method setup.
2Reliability
If all paths are drawn with equal probability through uniform edge initialization, then the search space exploration is unbiased initially, but superior architectures may be missed due to lack of preferential sampling
Solution Approach 1:
The patent performs preliminary uniform sampling of paths to evaluate and train multiple architectures initially. This preliminary phase allows the system to gather performance data from diverse architectures before committing to a final selection, ensuring both unbiased exploration and identification of superior architectures through subsequent probability adjustments.
Solution Approach 2:
The method implements feedback by using training results from sampled paths to adjust edge probabilities. Architectures that demonstrate superior performance during training provide feedback that increases the probability of sampling similar paths in the future, thereby systematically identifying and selecting high-quality architectures while maintaining initial exploration diversity.
Data Source
AI summary
A method for creating a machine learning system. The method includes: providing a directed graph including an input and an output node, each edge being assigned a probability that characterizes at which probability an edge is drawn. The probabilities are initially set to a value that paths are drawn at the same probability starting from the particular edge up to the output node.