Learnable Function Layer for Neural Network Channel Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for reducing the size of artificial neural networks, such as channel pruning and soft gating, face limitations in accuracy and computational efficiency, particularly in on-device platforms like smartphones and IoT devices, due to exponential comparison computations and incomplete weight learning.
Innovation Solution
An electronic device appends a learnable function layer to an existing artificial neural network, updates parameters through training, and uses the learnable function to eliminate channels, thereby reducing the network size while maintaining accuracy without exhaustive channel comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If channel pruning is performed by calculating absolute value sum of weights, then the channel size is reduced, but the pruning accuracy is insufficient because the channel with smallest sum is not necessarily the most unimportant channel
Solution Approach 1:
The patent changes the parameter used for channel selection from the simple sum of absolute weights to a more sophisticated criterion based on the learnable function output. By transforming the selection parameter from a crude metric to a refined one that better captures channel importance, the patent achieves more accurate pruning while reducing channel size.
Solution Approach 2:
The patent introduces a learnable function as an intermediary between the raw weight values and the channel selection decision. This intermediary layer processes the weight information and provides a more nuanced assessment of channel importance, enabling more accurate pruning decisions compared to direct weight sum comparison.
2Measurement precision
If all channel combinations are compared to find optimal pruning, then the pruning accuracy is improved, but the computational complexity becomes exponential
Solution Approach 1:
The patent employs a learnable function that automatically learns to identify important channels through training, eliminating the need for exhaustive combination comparisons. The system serves itself by developing an internal mechanism for channel evaluation that does not require external exhaustive search, thus reducing computational complexity while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary training of the learnable function on the full network before pruning. This preliminary action allows the function to learn channel importance patterns in advance, so that during actual pruning, simple function evaluation replaces complex exhaustive comparisons, significantly reducing computational burden.
3Stability of the object's composition
If soft gating with real number weights between 0 and 1 is used, then the channel elimination becomes smoother, but the weight learning is incomplete requiring additional regularization or annealing algorithms
Solution Approach 1:
The patent extracts the channel selection mechanism from the soft gating framework and implements it as a distinct learnable function layer. By separating the selection function from the weight multiplication approach, the patent achieves complete learning without needing additional regularization or annealing algorithms, while still maintaining smooth transition capabilities.
4Adaptability or versatility
If variational method with probability distribution is used for channel elimination, then the channel selection becomes more flexible, but repeated sampling is required increasing computational overhead
Solution Approach 1:
The patent replaces the expensive repeated sampling process with a single evaluation of the learnable function. The learnable function provides a deterministic yet flexible channel selection mechanism that captures the adaptability of probabilistic methods without requiring multiple samples, thus significantly improving computational efficiency.
Data Source
AI summary
An electronic device and a control method therefor are provided. The electronic device may comprise: a memory for storing at least one instruction; and a processor connected to the memory so as to control the electronic device, wherein the processor: by executing the at least one instruction, appends a second layer including a learnable function to a first layer in an artificial neural network including a plurality of layers; updates a parameter value included in the second layer by learning of the artificial neural network; acquires a function value by inputting the updated parameter value to the learnable function; and eliminates at least one channel among a plurality of channels included in the first layer on the basis of the acquired function value so as to achieve update to a third layer.