Neural Network Building Method with Random Operator Discarding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network architecture search methods based on first-order optimization face challenges in balancing convergence speed and search quality, leading to suboptimal performance due to the co-adaptation problem and Matthew effect, where operators with fewer parameters dominate, suppressing those with more parameters.
Innovation Solution
A method that initializes a search space with multiple operators, randomly discards some operators during training to ensure fairness, and updates building blocks to build a target neural network, ensuring all operators are expressed, thereby addressing the co-adaptation problem and Matthew effect.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If first-order optimization is used to speed up convergence, then convergence speed is improved, but search quality deteriorates due to operator co-adaptation and Matthew effect
Solution Approach 1:
The patent changes the parameter of operator selection probability by introducing a temperature parameter T. The selection probability is modified from uniform distribution to a distribution controlled by temperature, where higher temperature increases exploration and lower temperature increases exploitation. This allows the system to balance between converging quickly and maintaining search quality by adjusting the temperature schedule during training.
Solution Approach 2:
The patent introduces dynamic operator discarding where operators are randomly discarded during training with a certain probability. This dynamic mechanism prevents any single operator from dominating throughout the entire training process, allowing different operators to be expressed at different stages. The discarding probability can be adjusted dynamically to balance convergence speed and search quality.
2Speed
If all operators are simultaneously optimized, then convergence speed is improved, but operator expression fairness deteriorates leading to suboptimal architecture
Solution Approach 1:
The patent applies partial action by not optimizing all operators simultaneously with equal weight. Instead, operators are selectively optimized based on temperature-controlled probabilities and random discarding. This partial optimization approach prevents dominant operators from completely suppressing others, while still maintaining fast convergence by focusing computational resources on promising operators at different training stages.
Solution Approach 2:
The patent introduces periodic operator discarding where operators are randomly discarded at regular intervals during training. This periodic action ensures that no operator maintains dominance throughout the entire training process, allowing periodic re-evaluation and expression of previously suppressed operators. The periodic discarding mechanism balances convergence speed with operator expression fairness.
Data Source
AI summary
A neural network building method and apparatus are disclosed, and relate to the field of artificial intelligence. The method includes: initializing a search space and a plurality of building blocks, where the search space includes a plurality of operators, and the building block is a network structure obtained by connecting a plurality of nodes by using the operator; during training, in at least one training round, randomly discarding some operators, and updating the plurality of building blocks by using operators that are not discarded; and building a target neural network based on the plurality of updated building blocks. In the method, some operators are randomly discarded. This breaks association between operators, and overcomes a co-adaptation problem during training, to obtain a target neural network with better performance.