Neural Network Operator Pruning for Efficient Structure Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training neural network models, such as pruning, low-rank factorization, quantization, and knowledge distillation, fail to efficiently reduce the search space and resource overhead in optimizing model structures for resource-limited devices, leading to inaccurate and inefficient search results.
Innovation Solution
A method involving calculating the importance of candidate operators in a neural network model using measurable indicators, selecting and removing operators based on importance, and adjusting weight parameters to achieve a consecutive forward and back propagation process, thereby narrowing the search space and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If automatic search of all neural network model structures is performed, then the optimal model structure can be found, but the time and resource overhead becomes unbearable
Solution Approach 1:
The patent segments the neural network model structure into multiple candidate structures that can be independently trained and evaluated. Instead of searching through all possible structures simultaneously, the method divides the search space into manageable segments (candidate structures) that can be processed in parallel or sequence, reducing the overall search time while maintaining optimization accuracy.
Solution Approach 2:
The patent performs preliminary training of multiple candidate neural network structures before final selection. By pre-training these candidates and evaluating their performance in advance, the method identifies promising structures without requiring exhaustive search, thus reducing time overhead while preserving the ability to find optimal structures.
2Measurement precision
If exhaustive verification of all neural network model structures is performed, then the best performing model is identified, but the resource overhead becomes unbearable
Solution Approach 1:
The patent applies partial action by training and evaluating only a selected subset of candidate neural network structures rather than exhaustively verifying all possible structures. This partial verification approach maintains sufficient model performance evaluation accuracy while significantly reducing computational resource overhead by focusing resources on the most promising candidates.
3Reliability
If model structure search is performed without narrowing the search space, then comprehensive model evaluation is achieved, but time and resource overhead increases
Solution Approach 1:
The patent performs preliminary evaluation and filtering of candidate model structures to identify promising candidates before detailed verification. This preliminary action narrows the search space by eliminating clearly inferior structures early, maintaining search comprehensiveness for viable candidates while improving overall search efficiency through staged evaluation.
Solution Approach 2:
The patent enables the model structure search process to self-optimize by using performance feedback from preliminary evaluations to guide subsequent search directions. The system automatically identifies and focuses computational resources on promising candidate structures, reducing manual intervention and improving search efficiency while maintaining comprehensiveness through adaptive resource allocation.
Data Source
AI summary
The present disclosure provides a training method for the neural network model including a calculation step of calculating importance of candidate operators in the neural network model with respect to accuracy of a network output based on a measurable indicator, wherein the candidate operators in the neural network model include at least one of a first type of operator including a learnable parameter or a second type of operator not including a learnable parameter; a selection step of selecting a candidate operator from the neural network model based on the importance of the candidate operators; and an update step of removing a selected candidate operator from the neural network model and adjusting a weight parameter in the neural network model to obtain an efficient neural network model, wherein the efficient neural network model is a neural network model in which a forward propagation process and a back propagation process are consecutive.