Continuous RL Neural Architecture Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search methods, such as DARTS, suffer from limited search space exploration, bias towards specific architectures, instability, and discretization errors due to gradient-based optimization and supernet training approaches.
Innovation Solution
The implementation of a continuous reinforcement learning-based search algorithm that uses an actor-critic framework to explore the search space more effectively, allowing for increased batch sizes and accurate representation of architectures, and minimizing discretization errors by training a supernet once and reusing it for multiple operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If gradient-based search algorithm (DARTS) is used to speed up NAS, then search speed is improved, but search space exploration capability deteriorates
Solution Approach 1:
The patent replaces the gradient-based optimization mechanism (mechanical system) with a reinforcement learning-based search algorithm. The RL agent explores the search space by selecting architectures and receiving rewards based on performance, rather than following gradient descent paths. This substitution enables broader search space exploration while maintaining computational efficiency through the RL framework's ability to generalize across architecture configurations.
2Productivity
If supernet training is used to evaluate architectures, then evaluation efficiency is improved, but discretization error increases
Solution Approach 1:
The patent introduces an intermediary evaluation mechanism that bridges the supernet training approach and discrete architecture assessment. The RL agent evaluates candidate architectures by sampling from the supernet and using the observed performances to update its policy, rather than directly optimizing supernet weights. This intermediary approach reduces discretization error by maintaining the supernet's computational efficiency while accurately assessing discrete architecture performances through repeated sampling and reward-based learning.
3Speed
If gradient descent optimization is used to converge quickly, then convergence speed is improved, but search space coverage deteriorates
Solution Approach 1:
The patent implements dynamic exploration strategies where the RL agent adaptively balances exploitation of promising regions and exploration of new areas in the search space. Unlike static gradient descent that follows fixed gradient paths, the RL agent dynamically adjusts its architecture selection based on accumulated rewards and exploration bonuses, enabling both rapid convergence to high-performing architectures and comprehensive coverage of the search space through intelligent exploration.
Data Source
AI summary
A method and system for generating neural architectures to perform a particular task. An actor neural network, as part of a continuous action reinforcement learning (RL) agent, generates a randomized continuous actions parameters to encourage exploration of a search space to generate candidate architectures without bias. The continuous action parameters are discretized and applied to a search space to generate candidate architectures, the performance of which for performing the particular task is evaluated. Corresponding reward and state are determined based on the performance. A critic neural network, as part of the continuous action RL agent, learns a mapping of the continuous action to a reward using modified Deep Deterministic Policy Gradient (DDPG) with quantile loss function by sampling a list of top performing architectures. The actor neural network is updated with the learned mapping.