Neural Architecture Search Loss Function for Performance Collapse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search (NAS) methods face performance collapse and discrepancy issues due to aggregation of skip connections and over-parameterization, leading to inefficient training and suboptimal network design.
Innovation Solution
The implementation of a loss function that incorporates smooth maximum unit regularization and adjusts multiplicative and parametric connection weights to mitigate performance collapse, combined with processing only a portion of channels in each epoch for accelerated training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If continuous relaxation of candidates and one-step approximation of bi-level optimization are used in NAS, then the architecture search can be automated, but performance collapse occurs due to aggregation of skip connections
Solution Approach 1:
The patent changes the parameter representation from continuous relaxed weights to discrete operation selection. By using a discrete search space with operations like convolution, pooling, and skip connections as distinct choices rather than continuous weight values, the method avoids the performance collapse issue while maintaining automation through differentiable architecture search techniques.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of a supernet that acts as a mediator between the architecture search process and the final network derivation. The supernet contains all possible operations as candidate connections, and through controlled training and pruning, it enables automated search while preventing performance collapse by maintaining proper connection structures throughout the process.
2Adaptability or versatility
If all operations are used in weight-sum manner during supernet search, then the search space is fully explored, but the final derived network shows performance discrepancy due to over-parameterization
Solution Approach 1:
The patent extracts only the necessary operations from the over-parameterized supernet to create the final compact network. Through pruning operations that remove redundant connections and operations based on their contribution to performance, the method derives a precise final network that maintains search space exploration benefits while eliminating the performance discrepancy caused by over-parameterization.
Solution Approach 2:
The patent employs partial action by training the supernet with all possible operations initially, then selectively retaining only the essential operations for the final network. This approach allows full exploration of the search space during training while achieving precise performance in the final derived network by removing excessive parameters through systematic pruning.
3Ease of manufacture
If L1 or L2 metrics or weight-decay loss are used for supernet optimization, then the optimization process is simplified, but the performance of derived networks deteriorates
Solution Approach 1:
The patent changes the optimization approach from using L1/L2 regularization metrics to a discrete operation selection framework with a customized loss function. This new parameterization method evaluates architecture quality based on actual performance metrics and structured connection patterns, achieving both optimization simplicity and high derived network performance by focusing on operation-level decisions rather than weight-level regularization.
Data Source
AI summary
A system and a method are disclosed for neural architecture search. In some embodiments, the method includes: processing a training data set with a neural network during a first epoch of training of the neural network; computing a training loss using a smooth maximum unit regularization value; and adjusting a plurality of multiplicative connection weights and a plurality of parametric connection weights of the neural network in a direction that reduces the training loss.