Hierarchical Grouping in Differentiable Architecture Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing differentiable architecture search methods, such as DARTS and STACNAS, face issues like multicollinearity leading to unreasonable weight distributions and a gap in the first-stage selection, which affects the accuracy and stability of the searched network architectures.
Innovation Solution
A method for differentiable architecture search based on a hierarchical grouping mechanism, which involves selecting normal and reduction cells to form an initial search network, training it using one-level or two-level searches, and constructing a target network by optimizing control weight parameters and pruning operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If DARTS algorithm is used for differentiable architecture search, then search time is greatly reduced, but operations in search space have multicollinearity resulting in unreasonable weight distribution
Solution Approach 1:
The patent segments the search space by introducing a hierarchical grouping mechanism that divides operations into different groups (e.g., convolution groups, pooling groups, skip-connect groups). This segmentation prevents multicollinearity by ensuring operations within each group are structurally distinct, thereby resolving the weight distribution problem while maintaining search efficiency.
Solution Approach 2:
The patent applies local quality by assigning different weight initialization strategies and optimization parameters to different operation groups. Each group can have customized weight distributions and learning rates, allowing the algorithm to handle the unique characteristics of each operation type and avoid the uniform weight distribution problem in DARTS.
2Reliability
If STACNAS algorithm is used with first-stage search, then optimal operations are selected more stably, but a gap exists when only one operation is selected to represent group operation
Solution Approach 1:
The patent merges multiple operations within each group by maintaining all operations active simultaneously with group-level control weights. This combining approach preserves the full information of all operations in a group while still enabling stable selection through the hierarchical weighting mechanism, avoiding the information loss of selecting only one representative operation.
Solution Approach 2:
The patent creates a universal representation mechanism where each group of operations shares common control weights that can regulate the entire group's contribution. This multi-functional approach allows a single set of group weights to control multiple operations simultaneously, maintaining both stability and information completeness.
3Measurement precision
If DARTS algorithm performs network search, then search results may be satisfactory, but the searched network architecture is not identical to the network finally constructed
Solution Approach 1:
The patent performs preliminary action by conducting the search in a hierarchical manner where group-level structures are established first, followed by detailed operation optimization within each group. This preliminary structuring ensures that the searched architecture maintains consistency with the final constructed network, as the hierarchical framework is preserved throughout the process.
Solution Approach 2:
The patent implements feedback mechanisms at both group-level and operation-level during the search process. The hierarchical feedback ensures that changes in operation weights are consistent with group-level decisions, maintaining architecture consistency between the searched and final networks while still achieving high-quality search results.
Data Source
AI summary
A method for differentiable architecture search based on a hierarchical grouping mechanism includes: obtaining a target dataset; selecting a set number of normal cells and two reduction cells; enabling the reduction cells to be located at ⅓ and ⅔ of all cells, respectively, and then concatenating all cells to form an initial search network; training the initial search network to optimize a cost function to complete a one-level search or a two-level search, wherein in the one-level search, control weight parameters are shared by the inter-group operations and are shared by the intra-group operations among the cells; and in the two-level search, the control weight parameters are shared by the inter-group operations and are not shared by the intra-group operations, obtaining normal cells and reduction cells based on the one-level search or the two-level search; and constructing a target network using the normal cells and the reduction cells.