Neural Architecture Search Space Generation via Local Mutation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Neural Architecture Search (NAS) systems face challenges in efficiently designing optimal network architectures due to the complexity and size of the search space, relying heavily on human expertise and computation-intensive evaluation processes, which limits the discovery of novel and high-performing architectures.
Innovation Solution
A differentiable evolutionary framework is introduced to automate the design of search spaces, utilizing a local mutator model, reference directed acyclic graph (DAG) model, and differentiable scoring function to evolve the full search space into an optimal subspace, improving exploration capability and efficiency while avoiding sub-optimal solutions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the search space for neural network architecture is expanded to include more operations and configurations, then the potential for discovering high-performing architectures increases, but the computational cost and time required for evaluation increases significantly
Solution Approach 1:
The patent segments the search space into multiple independent components or modules, allowing the evaluation process to be divided into smaller, parallelizable tasks. This enables comprehensive search space coverage while reducing the sequential evaluation time by processing different segments concurrently.
Solution Approach 2:
The patent performs preliminary actions by pre-evaluating and caching results for common substructures or operations before the main search process. This allows the system to quickly reference previously computed results during the search, significantly reducing redundant evaluation time while maintaining thorough search space exploration.
2Manufacturing precision
If manual design expertise is used to guide the search process, then the quality of discovered architectures improves, but the extent of automation decreases
Solution Approach 1:
The patent introduces an intermediary system that bridges manual design expertise and automated search. This intermediary learns from expert-designed architectures and encodes their principles into automated evaluation criteria or search heuristics, allowing automated systems to benefit from human expertise without requiring manual intervention in each design iteration.
Solution Approach 2:
The patent implements self-service mechanisms where the system automatically learns and adapts design principles from data without requiring continuous manual guidance. The automated system improves its own performance by identifying patterns and optimizing architectures based on learned principles, reducing dependence on manual expertise while maintaining high architecture quality.
3Reliability
If computation-intensive evaluation processes are used to ensure thorough assessment of candidate architectures, then the reliability of performance estimation improves, but the productivity of the NAS system decreases
Solution Approach 1:
The patent applies partial action by performing computation-intensive evaluation only on the most promising candidate architectures identified through preliminary filtering. Instead of thoroughly evaluating all candidates, the system focuses computational resources on a selective subset, maintaining reliable performance estimation for critical candidates while improving overall productivity through reduced total evaluation burden.
Data Source
AI summary
A super-network comprising a plurality of layers may be generated. Each layer may comprise cells with different structures. A predetermined number of cells from each layer may be selected. A plurality of cells may be generated based on selected cells using a local mutation model, wherein the local mutation model comprises a mutation window for removing redundant edges from each selected cell. Performance of the plurality of cells may be evaluated using a differentiable fitness scoring function. The operations of the generating a plurality of cells using the local mutation model, the evaluating performance of the plurality of cells using the differentiable fitness scoring function and the selecting the subset of cells based on the evaluation results may be iteratively performed until the super-network converges. A search space for each layer may be generated based on a predetermined top number of cells with largest fitness scores after the super-network converges.