Subnetwork Sampling in Hypernetwork Topology
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In Neural Architecture Search (NAS), the efficiency of evaluating neural network substructures is low due to the need for additional training of Batch Normalization (BN) modules after subnetwork sampling, especially when output characteristics of different substructures are significantly different, leading to increased training time and costs.
Innovation Solution
A method for subnetwork sampling in a hypernetwork topology where each substructure includes a BN module corresponding to its closest upper layer, allowing direct selection and connection of these modules to add the substructure into the subnetwork without additional training, thereby eliminating the need for further training of the sampled subnetwork.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If each BN module learns a BN parameter for output characteristics of each substructure of its closest upper layer, then the BN module can handle different substructures, but when output characteristics of different substructures are greatly different, the BN module cannot reach an ideal learning state, requiring additional training time
Solution Approach 1:
The patent segments the BN module into multiple independent BN modules, where each BN module corresponds to a specific substructure of the upper layer. This segmentation allows each BN module to specialize in handling characteristics from a particular substructure, achieving ideal learning states without requiring additional training time for adaptation.
Solution Approach 2:
The patent applies local quality by making each BN module have specialized parameters tailored to its corresponding substructure. Instead of using a single BN module with generic parameters for all substructures, each BN module has locally optimized parameters that are specifically adapted to the characteristics of its associated substructure, thereby eliminating the need for additional training.
2Productivity
If a hypernetwork includes all search network structure spaces with shared parameters, then subnetwork sampling can be performed without retraining, but when subnetworks are sampled, additional training time is required to ensure BN modules reach ideal learning state
Solution Approach 1:
The patent performs preliminary action by pre-configuring multiple BN modules in the hypernetwork, each corresponding to a specific substructure of the upper layer. This preliminary setup ensures that when subnetworks are sampled, the appropriate BN modules are already in place with their parameters initialized, eliminating the need for additional training time to adapt BN modules to different substructures.
Solution Approach 2:
The patent introduces dynamics by making the selection of BN modules dynamic based on the sampled subnetwork structure. When a subnetwork is sampled, the system dynamically selects and activates the corresponding BN modules that match the sampled substructures, allowing the system to adapt to different subnetwork configurations without requiring retraining or additional training time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided are a method for subnetwork sampling and a method and device for building a hypemetwork topology. The method for subnetwork sampling is applicable to a hypemetwork topology. The hypemetwork topology includes n layers, each layer includes at least two substructures, and each substructure includes Batch Normalization (BN) modules in one-to-to correspondence with the substructures of a closest upper layer, n>0 and n being a positive integer. The method includes that: a substructure A(N) of the Nth layer is selected, 1>N≥n; a selected substructure A(N-1) of the (N-1)th layer is determined; a BN module C(B) in one-to-to correspondence with A(N-1) is determined from the substructure A(N); and the substructure A(N) is added into a subnetwork through the BN module C(B).