CNN Filter Pruning via Scaling Subnetwork
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face challenges with redundancy in filters, leading to impaired model generalization and increased computation costs, particularly in resource-constrained environments like embedded processors, where existing pruning methods are cumbersome and require multiple hyperparameters and fine-tuning steps.
Innovation Solution
The introduction of a scaling neural subnetwork within CNNs to automatically estimate filter importance during training, using a scale vector to quantify and remove redundant filters, thereby reducing the network size without accuracy loss, and eliminating the need for fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional filter pruning methods are used, then network size is reduced, but multiple hyperparameters and fine-tuning steps are required making the process cumbersome
Solution Approach 1:
The patent applies self-service by enabling the neural network to automatically evaluate filter importance and perform pruning without external intervention. The system uses internal mechanisms (filter importance evaluation based on magnitude, automated pruning criteria) to self-optimize the network structure, eliminating the need for manual hyperparameter tuning and multiple fine-tuning steps while reducing network size
2Productivity
If filters are removed to reduce network size, then computation cost decreases, but model generalization may be impaired
Solution Approach 1:
The patent applies preliminary action by evaluating filter importance before pruning occurs. The system pre-computes importance metrics based on filter magnitude and patterns, then uses these pre-evaluations to guide the pruning process. This ensures that only truly redundant filters are removed while preserving those critical for generalization, thus maintaining model performance while reducing computation cost
3Adaptability or versatility
If a large network is used to find complex functions, then capacity increases, but redundant parameters increase
Solution Approach 1:
The patent applies local quality by differentiating between filters based on their individual importance characteristics. Instead of uniform treatment, the system evaluates each filter's magnitude and patterns to assign different weights in the pruning process. Critical filters maintain full weight while redundant filters are pruned, allowing the network to maintain high function capacity through retained important filters while reducing overall parameter redundancy
Data Source
AI summary
An automated pruning technique is proposed for reducing the size of a convolutional neural network. A large-sized network is trained and then connections between layers are explored to remove redundant parameters. Specifically, a scaling neural subnetwork is connected to the neural network and designed to infer importance of the filters in the neural network during training of the neural network. Output from the scaling neural subnetwork can then be used to remove filters from the neural network, thereby reducing the size of the convolutional neural network.