Channel Scaling Layers for Selective Neural Network Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transfer learning in medical image interpretation often results in large, inefficient neural networks and reduced explainability due to the wholesale use of pre-trained models without selective channel pruning, which is data-intensive and hinders deployment on cloud or mobile devices and regulatory compliance.
Innovation Solution
The introduction of channel-scaling layers in deep neural networks, where scaling weights are trained to infer channel importance, allowing for selective channel removal and reducing the network size without altering original weights, using a computationally feasible approach that iteratively refines channel selection through backpropagation and L1 regularization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If wholesale use of pre-trained neural networks is applied, then model performance is maintained, but model size becomes unnecessarily large
Solution Approach 1:
The patent extracts only the essential feature channels from pre-trained neural networks by introducing channel-scaling layers that learn to identify and retain important channels while removing redundant ones. This selective extraction maintains model performance while significantly reducing model size for deployment on cloud or mobile devices.
Solution Approach 2:
The patent applies local quality by allowing different channels to have different scaling factors learned through channel-scaling layers. Each channel is individually evaluated and scaled based on its importance to the specific medical imaging task, rather than uniformly using all pre-trained channels, thus optimizing the balance between performance and size.
2Ease of manufacture
If bulk of pre-trained layers are used without detailed selections, then training simplification is achieved, but explainability is reduced
Solution Approach 1:
The patent extracts and retains only the most relevant feature channels for the specific medical imaging task by learning channel-specific scaling factors. This selective retention maintains training simplicity while improving explainability by focusing on the subset of channels that actually drive performance, making it easier to understand which features are important for medical diagnoses.
3Adaptability or versatility
If network pruning with fine-tuning iterations is applied, then channel selection is achieved, but data requirements increase
Solution Approach 1:
The patent performs preliminary channel selection by training channel-scaling layers on the target medical imaging dataset before final model deployment. This preliminary action identifies and retains important channels without requiring multiple iterative fine-tuning cycles, thereby reducing data requirements while maintaining the ability to select appropriate channels for the specific task.
Solution Approach 2:
The channel-scaling layers serve as an intermediary mechanism that enables channel selection without requiring extensive fine-tuning iterations. These layers learn to scale channel activations based on their importance, providing a data-efficient way to perform channel selection that avoids the need for multiple rounds of training with large datasets.
4Measurement precision
If network pruning iterations with fine-tuning are applied, then channel importance evaluation is improved, but reusability of selected channels decreases
Solution Approach 1:
The patent performs preliminary channel importance evaluation by training channel-scaling layers on the target dataset, which learns to identify important channels specific to that task. Once the scaling factors are learned and channels are selected, the same channel selection criteria can be reused for similar medical imaging tasks without requiring re-evaluation, thereby maintaining both precision and reusability.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for introducing channel-scaling layers in a deep neural network. A computer receives a pre-trained deep neural network including convolutional layers followed by respective ones of activation layers, adds channel-scaling layers after the respective ones of the activation layers, where each of the channel-scaling layers includes scaling weights. The computer trains the scaling weights in the channel-scale layers. The computer removes, in the convolutional layers, channels whose corresponding scaling weights are lower than a predetermined threshold. The computer removes the channel-scaling layers. In response to determining that at least one convergence criterion is met, the computer provides a finally trained deep neural network.