Neural Network Pruning via Parameter Importance Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of reducing the weight of neural networks through pruning requires a lengthy learning period, as deleting parameters gradually is necessary to minimize recognition accuracy deterioration, leading to a prolonged development and updating of recognition processing functions, especially when dealing with multiple types of recognition processing or environmental changes.
Innovation Solution
A learning processing device and method that identifies and prunes unimportant parameters from a neural network using a second learning dataset, allowing for the generation of a lightweight model in a shorter time by initializing the network with a pre-trained model and evaluating parameter importance through the Hessian or Fisher information matrix for efficient pruning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parameters are deleted gradually to minimize recognition accuracy deterioration, then recognition accuracy is preserved, but the learning period becomes long
Solution Approach 1:
The patent applies preliminary action by performing importance evaluation of parameters before the actual pruning process. The system calculates importance scores for all parameters using the evaluation unit, then uses these pre-calculated scores to guide the pruning process. This allows the system to identify which parameters to remove in advance, avoiding the need for repeated trial-and-error pruning cycles and significantly reducing the learning period while maintaining recognition accuracy.
2Loss of time
If a large number of parameters are deleted at a time, then the learning period is shortened, but recognition accuracy is largely deteriorated
Solution Approach 1:
The patent applies parameter changes by introducing an importance evaluation mechanism that transforms the pruning process from random or uniform parameter removal to selective removal based on calculated importance scores. The system changes the parameter selection criterion from arbitrary to data-driven, using importance evaluation results to determine which parameters can be safely removed. This allows for more aggressive pruning while maintaining recognition accuracy.
3Adaptability or versatility
If multiple lightweight models are prepared for different recognition processing types, then recognition processing versatility is improved, but the learning period and development time become long
Solution Approach 1:
The patent applies universality by creating a single pruning system with importance evaluation capability that can be applied to generate multiple different lightweight models from one base model. The importance evaluation unit can assess parameters for different recognition tasks, and the pruning unit can generate specialized lightweight models for each task type. This multi-functional approach allows the system to produce versatile model variants without requiring separate development processes for each, significantly reducing development and updating time.
Data Source
AI summary
A learning processing device and method achieves learning of a lightweight model that is completed in a short amount of time. The learning processing device obtains a new, second learning model from an existing first learning model. An input unit acquires a first learning model generated in advance by learning a first learning data set, and an unpruned neural network (hereinafter, NN). An important parameter identification unit uses the first learning model and the NN to initialize a NN to be learned, and uses a second learning data set and the initialized NN to identify a degree of importance of parameters in a recognition process of the initialized NN. A new model generation unit carries out a pruning process for deleting parameters which are not important from the initialized NN, thereby generating a second NN; and a learning unit uses the second learning data set to learn the second NN.