Neural Network Architecture Search via Flow Trajectories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Determining the optimal architecture for artificial neural networks is complex and resource-intensive, often resulting in high consumption of resources without ensuring the accuracy of finding the actually optimal architecture.
Innovation Solution
A method utilizing a directed graph to represent possible architectures, where nodes symbolize subsets of architectures, edges represent links, and flows are associated with edges, allowing for the optimization of trajectories based on rewards and cost functions to ascertain the optimal architecture without gradient estimation, thereby reducing resource consumption and increasing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If gradient-based methods are used to ascertain optimal architecture through repeated training and evaluation, then the architecture optimization can be automated, but the resource consumption increases significantly
Solution Approach 1:
The patent replaces the traditional gradient-based optimization mechanism (which requires repeated forward and backward passes through the network) with a flow-based approach. Flow trajectories are sampled through the computational graph without requiring gradient computation, thereby substituting the mechanical optimization process with a more efficient flow-based method that reduces computational overhead and resource consumption while maintaining automation.
2Measurement precision
If the number of iterations is increased to improve architecture search accuracy, then the optimal architecture can be found more precisely, but the resource consumption and time required increase
Solution Approach 1:
The patent performs preliminary actions by pre-defining the search space and architecture modules before the actual optimization process. By preparing the computational graph structure and possible architecture configurations in advance, the method enables more accurate architecture search with fewer iterations, as the flow-based approach can efficiently explore the pre-prepared search space without requiring extensive repeated training.
3Extent of automation
If traditional NAS methods are used to optimize architecture, then automation is achieved, but the method becomes susceptible to noise and requires more resources
Solution Approach 1:
The patent introduces flow trajectories as an intermediary between the architecture search process and the evaluation metric. Instead of directly evaluating architecture performance through repeated training (which is noisy and resource-intensive), the flow-based approach uses flow trajectories as a mediator to estimate architecture quality, providing a more stable and noise-resistant optimization process that maintains automation while improving reliability.
Data Source
AI summary
A method for ascertaining an optimal architecture of an artificial neural network. The method includes: ascertaining the optimal architecture of the artificial neural network by repeatedly ascertaining a trajectory from the initial node to a terminal node based on the defined strategy, determining a reward for the ascertained trajectory, determining a cost function for the ascertained trajectory based on the ascertained reward for the trajectory and the flows associated with the edges along the trajectory, and respectively updating the flows associated with the edges along the trajectory, based on the cost function until an ascertained trajectory fulfills a termination criterion for the architecture search, wherein the trajectory that fulfills the termination criterion represents the optimal architecture.