Self-Pruning Neural Networks Using Learnable Weight Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep-Learning Neural Networks (DNNs) face high computational demands, particularly in handheld devices, due to the complexity of determining optimal weight values across multiple layers, leading to inefficient pruning methods that are time-consuming and result in significant accuracy loss.
Innovation Solution
An analytic threshold function is applied during back-propagation to prune weights, optimizing the number of non-zero weights by setting thresholds based on a differentiable function, with parameters α and β adjusted to achieve sharper edges and wider widths, reducing computational burden through fewer MAC operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If iterative pruning techniques are used to reduce weights, then the number of non-zero weights is reduced, but the process becomes time-consuming and requires multiple re-training iterations
Solution Approach 1:
The patent changes the parameter of the threshold function from a fixed value to a learnable parameter that is optimized during back-propagation. This allows the pruning threshold to be automatically adjusted based on the network's performance, eliminating the need for iterative manual tuning and re-training while achieving optimal sparsity.
Solution Approach 2:
The threshold function is designed to be self-adjusting through back-propagation, where the pruning parameters are automatically optimized by the network itself during training. This self-service mechanism eliminates the need for external iterative intervention and manual re-training iterations.
2Device complexity
If greedy layer-wise thresholding is applied to prune weights, then computational complexity is reduced, but overall system accuracy is significantly lost
Solution Approach 1:
The patent introduces feedback through back-propagation, where the impact of pruning on network accuracy is continuously evaluated and used to adjust the threshold function parameters. This feedback loop ensures that pruning is performed in a way that minimizes accuracy loss while maintaining computational efficiency.
Solution Approach 2:
The threshold function transitions from a static, fixed threshold to a dynamic, learnable function that adapts during training. This dynamic approach allows the pruning strategy to evolve based on the network's learning progress, maintaining accuracy while reducing complexity.
3Reliability
If more layers and weights are maintained in the neural network, then accuracy is improved, but computational demand increases making deployment in handheld devices difficult
Solution Approach 1:
The patent changes the parameter of sparsity control from a fixed post-training adjustment to a learnable parameter optimized during training. This allows the network to achieve optimal accuracy with fewer non-zero weights, reducing computational demand and energy consumption for deployment in resource-constrained handheld devices.
Data Source
AI summary
A technique to prune weights of a neural network using an analytic threshold function h(w) provides a neural network having weights that have been optimally pruned. The neural network includes a plurality of layers in which each layer includes a set of weights w associated with the layer that enhance a speed performance of the neural network, an accuracy of the neural network, or a combination thereof. Each set of weights is based on a cost function C that has been minimized by back-propagating an output of the neural network in response to input training data. The cost function C is also minimized based on a derivative of the cost function C with respect to a first parameter of the analytic threshold function h(w) and on a derivative of the cost function C with respect to a second parameter of the analytic threshold function h(w).