Neural Architecture Optimization via Subgraph Mutation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural architecture search (NAS) methods face limitations in generalizability, transferability, and efficiency, relying heavily on expert knowledge and being hardware-dependent, with limited ability to optimize existing neural networks and leverage knowledge across different search spaces.
Innovation Solution
The Automatic Graph Optimization (AutoGO) method uses a computational graph representation to iteratively mutate and optimize neural architectures, leveraging a database of useful subgraphs and mixed-integer linear programming for resolution propagation, enabling the combination of information from different search spaces and optimizing performance and resource metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual trial-and-error process is used for neural network design, then expert knowledge can be applied, but the design process is time-consuming and lacks automation
Solution Approach 1:
The system performs self-service by automatically searching and optimizing neural network architectures without requiring manual expert intervention. The neural architecture search algorithm autonomously explores the search space, evaluates candidate architectures, and selects optimal configurations, enabling the system to design its own neural network structures efficiently.
Solution Approach 2:
The system performs preliminary action by pre-defining a search space of possible neural network architectures and pre-establishing evaluation metrics before the actual search begins. This preparation phase includes setting up the computational graph, defining operation types, and configuring hyperparameters, which enables the automated search to proceed efficiently without ad-hoc adjustments during the design process.
2Reliability
If extensive search of neural architecture space is performed, then optimal architecture can be found, but the computational effort and time required increase exponentially
Solution Approach 1:
The system segments the neural architecture search process into distinct components: computational graph representation, subgraph identification, operation type assignment, and architecture assembly. By dividing the search space into manageable segments (subgraphs) and independently optimizing each, the system reduces the overall computational complexity while maintaining the ability to find high-quality architectures through systematic exploration of segmented search spaces.
3Reliability
If hardware-dependent optimization is performed, then performance on specific hardware can be improved, but the solution lacks generalizability to different hardware platforms
Solution Approach 1:
The system achieves universality by designing a hardware-agnostic computational graph representation that can be executed on multiple hardware platforms. The neural architecture search process evaluates architectures based on performance metrics that are independent of specific hardware implementations, allowing the same optimized architecture to be deployed across different hardware platforms (GPUs, TPUs, CPUs, edge devices) without re-optimization, thus providing both high performance and broad adaptability.
Data Source
AI summary
A computerized method has the steps of: generating an input computational graph (CG) for representing a neural architecture, and performing a plurality of optimization steps for at least one iteration to obtain a modified neural architecture represented by a modified CG, for obtaining a neural network for use in one or more computing devices. The optimization steps include: identifying one or more subgraphs from the input CG, obtaining the modified CG by replacing the identified one or more subgraphs with one or more replacement subgraphs, respectively, evaluating one or more metrics of a modified neural-network architecture represented by the modified CG, and based on the evaluation, selecting the modified CG or the input CG. When the optimization steps are performed for multiple iterations, the selected CG obtained in one iteration is used as the input CG for a next iteration.