Channel Pruning in Convolutional Neural Networks via Cost Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant computational resources due to their complex structure, and conventional methods for reducing complexity, such as sparsifying model weights, often break down in nonconvex learning problems, leading to performance degradation without clear guidelines for thresholding model weights or targeting specific layers for pruning.
Innovation Solution
The method involves determining a cost metric for each channel in convolution layers to identify and prune channels that contribute minimally to the information flow, using channel-scaling coefficients and iterative-thresholding algorithms to update weights, thereby reducing architectural complexity without significant performance impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks with many layers and channels are used to improve performance, then model accuracy and capability are improved, but computational resources (computation cycles, memory, storage space) increase supralinearly
Solution Approach 1:
The patent extracts and removes redundant channels from convolutional layers based on importance scoring. By identifying and eliminating channels that contribute minimally to the output, the method reduces computational resources while preserving the essential functionality and performance of the deep neural network.
Solution Approach 2:
The patent applies different treatment to different channels within the same layer by computing individual importance scores for each channel. This local differentiation allows the network to maintain high-quality processing in important channels while reducing or removing resources allocated to less important channels, resolving the contradiction between overall performance and resource usage.
2Productivity
If conventional sparsification methods are applied to reduce complexity, then computational efficiency is improved, but performance degradation occurs due to breakdown in nonconvex learning problems
Solution Approach 1:
Instead of applying fixed thresholding or norm penalties to model weights, the patent changes the approach by computing data-driven importance scores for each channel based on the actual training data distribution. This parameter change from fixed sparsification criteria to adaptive importance scoring allows the method to maintain performance while improving computational efficiency in nonconvex learning problems.
Solution Approach 2:
The patent incorporates feedback from the training data into the channel selection process by computing importance scores that reflect the actual contribution of each channel to the learning objective. This feedback mechanism ensures that channels removed for efficiency do not critically impact performance, as their importance is continuously evaluated based on the training data.
3Use of energy by moving object
If layers are pruned to reduce complexity, then resource consumption is reduced, but insufficient information is available to target specific layers without sacrificing performance
Solution Approach 1:
The patent changes from no channel-level analysis to comprehensive channel-level importance scoring. By computing importance scores for individual channels rather than treating entire layers uniformly, the method provides sufficient information to precisely target specific channels for pruning, thereby reducing resource consumption without sacrificing performance.
Data Source
AI summary
The architectural complexity of a neural network is reduced by selectively pruning channels. A cost metric for a convolution layer is determined. The cost metric indicates a resource cost per channel for the channels of the layer. Training the neural network includes, for channels of the layer, updating a channel-scaling coefficient based on the cost metric. The channel-scaling coefficient linearly scales the output of the channel. A constant channel is identified based on the channel-scaling coefficients. The neural network is updated by pruning the constant channel. Model weights are updated via a stochastic gradient descent of a training loss function evaluated on training data. The channel-scaling coefficients are updated via an iterative-thresholding algorithm that penalizes a batch normalization loss function based on the cost metric for the layer and a norm of the channel-scaling coefficients. When the layer is batch normalized, the channel-scaling coefficients are batch normalization scaling coefficients.