Neural Network Pruning via L2 Regularization and Inactive Node Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks require significant configuration to achieve high performance, making them inefficient for operation in edge systems where size reduction is necessary.
Innovation Solution
A learning method that optimizes neural networks by updating weight coefficients to minimize an objective function combining a basic loss function and an L2 regularization term, and subsequently identifies and removes inactive nodes and channels to reduce network size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the neural network configuration is increased to achieve high performance, then the recognition accuracy is improved, but the network size increases making it unsuitable for edge system deployment
Solution Approach 1:
The patent extracts and removes inactive nodes and channels from the neural network through systematic identification and deletion. The specifying unit identifies inactive components based on weight coefficient characteristics, and the deleting unit removes them to reduce network size while preserving the active components that contribute to recognition accuracy.
Solution Approach 2:
The patent applies different treatment to different parts of the neural network based on their activity status. Active nodes and channels are preserved to maintain recognition accuracy, while inactive nodes and channels are removed to reduce network size. This differential approach allows the network to achieve edge system suitability without compromising overall performance.
2Device complexity
If the neural network configuration is reduced to suit edge systems, then the network size is decreased, but the recognition accuracy deteriorates
Solution Approach 1:
The patent performs preliminary identification of inactive nodes and channels before removing them from the network. The specifying unit analyzes weight coefficients to determine which components are inactive, ensuring that only truly redundant elements are removed. This preliminary assessment prevents the removal of components that would contribute to recognition accuracy.
Solution Approach 2:
The patent uses feedback from the objective function evaluation to guide the network optimization process. The specifying unit monitors changes in recognition accuracy as nodes and channels are removed, and adjusts the deletion process accordingly to maintain performance thresholds while achieving size reduction.
3Reliability
If weight coefficients are updated using L2 regularization to minimize the objective function, then the network generalization is improved, but the computational complexity increases
Solution Approach 1:
The patent changes the parameter representation by identifying and removing inactive nodes and channels, which have weight coefficients close to zero. This parameter change simplifies the network structure while maintaining the L2 regularization effect on active components, reducing computational complexity without sacrificing generalization capability.
Data Source
AI summary
According to an embodiment, a learning method of optimizing a neural network, includes updating and specifying. In the updating, each of a plurality of weight coefficients included in the neural network is updated so that an objective function obtained by adding a basic loss function and an L2 regularization term multiplied by a regularization strength is minimized. In the specifying, an inactive node and an inactive channel are specified among a plurality of nodes and a plurality of channels included in the neural network.