Neural Network Pruning via Sensitivity Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network pruning methods fail to achieve structured sparsity, leading to inefficient compression and increased memory access costs, particularly in resource-constrained devices, as they independently minimize non-zero parameters for each neuron without preserving network structure.
Innovation Solution
A method that jointly promotes parameters towards zero using a neuronal sensitivity regularization function, allowing for structured simplification of neural networks while maintaining learning task performance, by driving less sensitive neurons and their connections towards zero.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing pruning methods independently minimize non-zero parameters for each neuron, then the number of non-zero parameters is reduced, but the network structure is compromised and memory access costs increase
Solution Approach 1:
The patent segments the pruning process into two distinct stages: first applying L1 regularization to promote sparsity at the parameter level, then applying group Lasso regularization to enforce structured sparsity at the neuron level. This segmentation allows each regularization term to operate at its appropriate granularity, achieving both parameter reduction and structural preservation
Solution Approach 2:
The patent changes the regularization parameters by introducing a composite loss function that combines L1 and group Lasso terms with controllable weights. By adjusting the regularization strength parameters λ1 and λ2, the system can control the degree of sparsity and structural preservation independently, resolving the contradiction between parameter reduction and structure maintenance
2Quantity of substance
If existing pruning methods independently minimize non-zero parameters for each neuron, then the number of non-zero parameters is reduced, but memory access costs and processing overhead increase significantly
Solution Approach 1:
The patent segments the sparsity enforcement into hierarchical levels: parameter-level sparsity through L1 regularization and neuron-level structured sparsity through group Lasso. This segmentation creates a two-tiered sparsity structure where entire neurons or groups are pruned together, enabling more efficient memory access patterns compared to random parameter pruning
Solution Approach 2:
The patent extracts and removes entire neurons or groups of neurons that are deemed insignificant based on the group Lasso regularization. By taking out complete functional units rather than individual parameters, the system achieves better memory access efficiency and reduces processing overhead while maintaining the essential network structure
3Quantity of substance
If the number of parameters is reduced through pruning, then model compression is achieved, but learning task performance may be affected
Solution Approach 1:
The patent incorporates feedback mechanisms through the regularization terms that continuously guide the pruning process. The L1 and group Lasso regularization terms provide ongoing feedback during training to preserve important parameters and neurons, ensuring that performance is maintained while achieving compression. The feedback is embedded in the loss function that guides gradient descent
Solution Approach 2:
The patent applies preliminary regularization constraints during the training phase before the actual pruning is finalized. By pre-training with L1 and group Lasso regularization, the network learns a sparse structure that maintains performance, and then this pre-structured sparsity is used to guide the final pruning decisions, ensuring performance is preserved
Data Source
AI summary
The present invention relates to a method for pruning a neural network comprising a plurality of neurons, said method comprising: an initialization phase, wherein input information is fetched comprising at least parameters ({wni,bni}) related to said neural network and a dataset (D) representative of a task that said neural network has to deal with, wherein said parameters ({wni,bni}) comprising a weights vector (wni) and/or a bias (bni) related to at least one neuron of said plurality of neurons; a regularization phase, wherein said neural network is trained according to a training algorithm by using said dataset (D); a thresholding phase, wherein an element (wnij) of said weights vector (wni) is put at zero when its absolute value is below a given threshold (T).