Meta-learning Graph Networks for Neural Architecture Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Neural Architecture Search (NAS) methods are inefficient in finding optimal neural network architectures for new tasks, as they typically restart the optimization process from scratch for each task, failing to leverage the structure of the lattice and neighborhood relations between related tasks, leading to prolonged processing times and suboptimal performance.
Innovation Solution
The use of meta-learning to predict lattice edge weights, leveraging neighborhood relations in the space of deep net architectures, allowing for faster convergence and better architecture discovery for new tasks by repurposing edge weights from previously trained models, thus enabling quicker and more efficient NAS.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Neural Architecture Search restarts optimization from scratch for each task, then it can find task-specific optimal architectures, but it leads to prolonged processing times and increased computing resource consumption
Solution Approach 1:
The patent pre-trains a graph network model on a source task to learn the mapping between graph features and architecture performance. This preliminary action enables the model to make informed predictions for new tasks without restarting optimization from scratch, thus reducing processing time while maintaining task-specific optimality through adaptive fine-tuning
Solution Approach 2:
The patent copies the learned graph network model parameters from a source task to a target task. By replicating the knowledge gained from optimizing architectures for one task and adapting it to another, the system avoids redundant computation while preserving task-specific performance requirements through subsequent fine-tuning
2Reliability
If Neural Architecture Search uses traditional methods for each task independently, then it ensures task-specific optimization, but it fails to leverage neighborhood relations between related tasks
Solution Approach 1:
The patent creates a universal graph network model that can handle multiple tasks by learning common patterns in the architecture search space. The model serves multiple functions: it can be trained on source tasks and applied to target tasks, leveraging neighborhood relations while adapting to task-specific requirements through fine-tuning
Solution Approach 2:
The graph network model acts as an intermediary between source task knowledge and target task requirements. It mediates the transfer of optimization insights by learning to predict architecture performance based on graph features, enabling efficient adaptation to new tasks while maintaining task-specific optimality
3Speed
If meta-learning is used to predict edge weights from previously trained models, then convergence speed improves, but the model complexity increases
Solution Approach 1:
The patent applies local quality by focusing the graph network model's predictions specifically on edge weights in the architecture search space, rather than attempting to predict all architecture parameters. This localized approach accelerates convergence while keeping the model complexity manageable by concentrating computational resources on the most critical predictive features
Data Source
AI summary
One or more embodiments of the disclosure include systems and methods that use meta-learning to learn how to optimally find a new neural network architecture for a task using past architectures that were optimized for other tasks, including for example tasks associated with autonomous, semi-autonomous, assisted, or other driving applications. A computer implemented method of the disclosure includes configuring a search space lattice comprising nodes representing operator choices, edges, and a maximum depth. The method includes defining an objective function. The method further includes configuring a graph network over the search space lattice to predict edge weights over the search space lattice. The method also includes alternating optimization between (1) weights of the graph network, to optimize the objective function over a validation set, and (2) weights corresponding to nodes of the search space lattice that are randomly initialized or configured using previously trained paths in the search space lattice.