Neural Network Channel Pruning via Intermediary Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network reduction technologies face challenges in reducing weighting filters in units of channels without deteriorating the performance of the neural network, as deletion of channel data can lead to loss of information contributing to network performance.
Innovation Solution
A neural network reduction device and method that involves generating a third neural network by connecting a second neural network as an additional layer to a convolutional layer, training the network, selecting and deleting redundant channels based on feature values, and correcting weight parameters using singular value decomposition or multiply-accumulate operations, followed by retraining to maintain network performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If channel data is deleted to reduce weighting filters, then the number of parameters is reduced, but information contributing to network performance is lost
Solution Approach 1:
The patent applies preliminary action by adding the second neural network as an additional layer before performing channel pruning. This preliminary addition allows the system to learn and capture important information from channels that will subsequently be pruned, thereby preventing information loss when channels are removed to reduce the number of parameters
Solution Approach 2:
The second neural network acts as an intermediary that bridges the original channels and the final pruned structure. By inserting this intermediate layer, the system can transfer and preserve important channel information before deletion, mediating between the need to reduce parameters and the need to maintain performance
2Device complexity
If weighting filters are reduced in units of channels, then the neural network size is reduced, but network performance deteriorates
Solution Approach 1:
The patent performs preliminary training with the second neural network added before channel pruning. This preliminary action allows the network to learn important features and relationships that would otherwise be lost during channel reduction, enabling aggressive pruning while maintaining performance
Solution Approach 2:
The patent changes parameters by introducing the second neural network with its own weight parameters and by modifying the training process to include this additional layer. These parameter changes enable the system to achieve better compression ratios while maintaining performance by learning more efficient representations
Data Source
AI summary
There is provided a neural network reduction device including: an input section configured to acquire a first neural network including a plurality of processing layers that includes at least one convolutional layer; a generation section configured to generate a third neural network in which at least one second neural network is connected as an additional layer to a stage subsequent to the convolutional layer included in the first neural network; a learning section configured to train the third neural network; a correction section configured to select, as a redundant channel, a channel whose feature value calculated on a basis of a weight parameter of the second neural network establishes a predetermined relationship with a predetermined threshold, delete the redundant channel from the convolutional layer, and correct a weight parameter of the convolutional layer after deletion of the redundant channel on a basis of at least a portion of the weight parameter of the second neural network; a relearning section configured to retrain the third neural network on a basis of correction of the weight parameter of the convolutional layer after the deletion of the redundant channel; and an output section configured to output the retrained third neural network.