Neural Network Pruning with Saliency Feedback for Reliable Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pruning techniques for neural networks are labor-intensive, require expert configuration, and can erroneously identify non-redundant structures as redundant, leading to degraded performance or inoperability.
Innovation Solution
A structured, iterative pruning method using saliency analysis to identify and project redundant groups to an origin point, transferring information to retained groups, with safeguards to prevent erroneous removal, allowing for controlled and reliable reduction of trainable parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing pruning techniques are used to reduce neural network size, then the number of trainable parameters is reduced, but the reliability of the neural network is compromised due to erroneous identification of non-redundant structure as redundant
Solution Approach 1:
The patent implements a feedback mechanism by repeatedly assessing the saliency of candidate redundant groups as they are projected to the origin point. This multi-assessment feedback loop allows the system to detect when a group's saliency increases during projection, indicating it should not be pruned, thereby preventing erroneous removal of non-redundant structure while maintaining parameter reduction.
Solution Approach 2:
The patent performs preliminary training of the neural network before the pruning process begins. This preliminary action ensures the network is properly initialized and functional before parameter reduction starts, establishing a baseline for reliable saliency assessment and preventing premature pruning that could compromise network reliability.
2Reliability
If expert configuration is required to set hyper-parameters for pruning, then the pruning process can be controlled, but the ease of operation is reduced due to labor-intensive setup
Solution Approach 1:
The patent implements self-service by using automated saliency assessment that dynamically identifies redundant groups based on their contribution to network functions. The system automatically determines which groups to prune through iterative saliency evaluation during the projection process, eliminating the need for expert configuration of hyper-parameters while maintaining reliable control over the pruning process.
3Quantity of substance
If aggressive pruning is applied to maximize parameter reduction, then the neural network size is minimized, but the performance of the neural network is degraded
Solution Approach 1:
The patent replaces traditional mechanical pruning approaches with a saliency-based assessment mechanism. Instead of blindly removing parameters or using fixed thresholds, the system dynamically evaluates the functional contribution of each parameter group through saliency analysis during projection, substituting rigid mechanical removal with intelligent, performance-aware selection that prevents degradation while maximizing parameter reduction.
Data Source
AI summary
A technique prunes an original neural network over plural pruning periods to reduce a number of groups of trainable parameters in the original neural network by a target number (K) of groups. The technique leverages saliency analysis to identify redundant groups and to-be-retained (important) groups. The pruning is performed by successively projecting the redundant groups to an origin point and successively transferring information contained in the redundant groups to the to-be-retained groups. In some implementations, the pruning also identifies a final set of redundant groups based on plural assessments of saliency of candidate redundant groups, as the candidate redundant groups are projected to the origin point. This aspect operates as a safeguard, reducing the risk that the pruning will degrade the performance of the neural network by erroneously removing non-redundant structure of the original neural network.