Neural Network SynFlow Pruning to Prevent Layer Collapse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network pruning methods at initialization suffer from layer collapse, leading to sudden drops in accuracy and requiring extensive computational resources, especially when identifying sparse subnetworks known as 'winning tickets', which are not efficiently addressed by current algorithms.
Innovation Solution
The SynFlow method employs an iterative synaptic flow pruning process that assigns scores based on a novel loss function, ensuring positive and conservative synaptic saliency scores, and iteratively evaluates these scores to avoid layer collapse, achieving maximal critical compression without requiring training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If iterative training and pruning cycles are used to identify winning tickets, then sparse subnetworks can be found that match original network accuracy, but computational costs increase significantly
Solution Approach 1:
The patent applies preliminary action by performing pruning at network initialization before training begins. The method computes gradient norms at initialization to score parameters, then prunes based on these scores without requiring iterative training cycles. This preliminary scoring and pruning approach identifies winning tickets upfront, avoiding the computationally expensive iterative training-pruning cycles of conventional methods while still achieving sparse subnetworks that match original network accuracy.
2Productivity
If single-shot pruning at initialization is used, then computational costs are reduced, but layer collapse occurs rendering the network untrainable
Solution Approach 1:
The patent applies feedback by using gradient norm information to guide the pruning process. Specifically, it computes gradient norms at initialization, uses these to score parameters, prunes based on scores, then re-computes gradient norms on the pruned network and iterates. This feedback loop ensures that pruning decisions are based on actual gradient flow information, preventing layer collapse while maintaining computational efficiency compared to iterative training approaches.
Solution Approach 2:
The patent applies dynamics by making the pruning process adaptive and iterative rather than static. The method dynamically re-computes gradient norms after each pruning step and adjusts subsequent pruning decisions based on the changed network structure. This dynamic approach allows the algorithm to respond to the evolving network state, preventing layer collapse while maintaining computational efficiency.
3Quantity of substance
If conventional pruning algorithms are used after training, then network compression is achieved, but training cost reduction is not realized
Solution Approach 1:
The patent applies preliminary action by performing all pruning operations at network initialization before training begins. By computing gradient norms at initialization and pruning based on these scores upfront, the method achieves network compression prior to training. This eliminates the need for post-training pruning, thereby realizing full training time reduction while achieving the desired parameter reduction.
Data Source
AI summary
A system and method to prune parameters of a neural network at initialization uses iterative conserving synaptic flow that saves time, memory and energy both during training and at test time of the neural network. The result of the disclosed pruning system and method are highly sparse trainable subnetworks at initialization, without training and without ever looking at the data (a data agnostic pruning system and method). The pruning system and method preserves the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint.