Supernet Learning with Complexity-Based Subnet Rate Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural architecture search methods, including one-shot and few-shot learning, face inefficiencies due to weight interference and inaccurate performance predictions in large supernet structures, leading to suboptimal neural network selection.
Innovation Solution
A supernet learning apparatus and method that dynamically adjusts the learning rate based on the complexity of each subnet extracted from the supernet, allowing for fair and accurate performance comparisons during neural architecture search.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all subnets in a large supernet share weights through one-shot learning, then learning efficiency is improved, but weight interference occurs causing inaccurate performance predictions
Solution Approach 1:
The patent divides the large supernet into multiple smaller sub-supernets based on subnet complexity. Each sub-supernet contains subnets with similar complexity levels, which reduces weight interference between subnets of different complexities. This segmentation allows for more accurate performance predictions while maintaining learning efficiency through weight sharing within each sub-supernet.
2Ease of operation
If learning is performed under the same conditions for all extracted subnets, then learning equality is provided for comparison, but subnets with different complexities cannot be fairly compared
Solution Approach 1:
The patent applies different learning rates to subnets based on their complexity characteristics. Subnets with higher complexity are assigned different learning rates compared to simpler subnets, allowing each subnet to learn at an appropriate pace for its complexity level. This enables fair comparison of neural architectures by accounting for complexity differences while maintaining operational simplicity.
3Adaptability or versatility
If the supernet is constructed to include all candidate neural networks, then the search space is fully covered, but the supernet size becomes very large causing computational burden
Solution Approach 1:
The patent segments the comprehensive search space into multiple sub-supernets based on subnet complexity. Each sub-supernet contains a manageable subset of candidate neural networks with similar complexity characteristics. This segmentation maintains full search space coverage while reducing the computational burden of training and evaluating each individual subnet, as weight sharing occurs within smaller sub-supernets rather than across the entire large supernet.
Data Source
AI summary
The disclosed embodiment provides a supernet learning apparatus and method that can accurately compare each subnet based on performance when searching for a neural architecture by performing learning by adjusting the learning rate according to the complexity of each subnet extracted from the supernet, by performing the steps of analyzing the complexity of a subnet repeatedly extracted from the supernet, and setting a learning rate that is dynamically variable according to the number of learning repetitions based on the complexity analyzed in the extracted subnet, learning the extracted subnet using the learning rate that is set to be variable according to the number of learning repetitions, and merging the trained subnet into the supernet to obtain a trained supernet.