Supermodel LORA Architecture Search to Prevent DARTS Collapse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
DARTS neural architecture search methods frequently fail to find optimal network architectures due to a tendency towards trivial operations, leading to performance degradation and requiring computationally intensive tuning.
Innovation Solution
Integrate Low Rank Adaptation (LORA) modules into a pre-trained supermodel, allowing operations to remain frozen during training, and utilize early pruning to stabilize the search space, thereby improving the DARTS approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DARTS trains the supermodel by jointly optimizing architecture weights and model weights, then the optimal network architecture can be discovered, but the model tends to perform trivial operations such as skip connections leading to performance degradation
Solution Approach 1:
The patent segments the weight parameters into two distinct sets: architecture weights (α) that control operation selection and model weights (β) that control operation execution. This segmentation allows independent optimization of architectural decisions from operational parameters, preventing the coupling that leads to trivial operation collapse while maintaining the ability to discover optimal architectures.
Solution Approach 2:
The patent changes the parameterization approach by introducing architecture weights as separate learnable parameters that gate operation selection through softmax functions. This parameter change transforms the discrete architecture search into a continuous optimization problem while maintaining training stability through proper gradient flow design that prevents collapse to trivial operations.
2Reliability
If training is stopped based on gradient analysis to mitigate trivial operations, then performance degradation is reduced, but further exploration of the search space is prevented
Solution Approach 1:
The patent implements continuous feedback through the architecture weight optimization process, where the softmax-based architecture weights provide real-time guidance on operation importance. This feedback mechanism allows the model to adaptively adjust operation selection during training without requiring early stopping, enabling both stability and continued exploration of the search space through gradient-based updates.
3Reliability
If a regularization term is added to increase correlation between architecture weights, then the DARTS error mode is mitigated, but performance losses occur and computational tuning is required
Solution Approach 1:
The patent enables the model to self-regulate architecture weight correlation through the inherent structure of the softmax function and gradient-based optimization. The architecture weights automatically develop appropriate correlations during training without requiring external regularization terms, as the optimization process naturally encourages consistent architectural patterns across layers while maintaining the ability to adapt to task-specific requirements.
Data Source
AI summary
A method for extracting an optimal network architecture for solving a target task. The method includes: providing a supermodel pre-trained based on labeled training data for solving the target task, wherein the supermodel includes a plurality of pre-trained operations; adding at least one LORA module to at least one of the operations of the supermodel, wherein the LORA modules in each case include trainable weights; training the pre-trained supermodel by training the respective weights of the relevant LORA module, until a certain training criterion is reached, wherein the at least one of the operations of the supermodel remains unchanged during the training of the weights; extracting an optimal network architecture for solving the target task from the trained supermodel based on the architecture weights; and providing the extracted optimal network architecture for solving the target task.