Neural Network Pruning via Self-Distillation and Weight Reintroduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for pruning neural networks often result in a significant loss of accuracy and are limited in the number of weights they can remove before impacting performance, leading to inefficient models with larger memory footprints.
Innovation Solution
The method employs a novel approach of self-distillation from a dense model to a sparse model, using knowledge distillation and weight reintroduction to prune and boost weights, allowing for more extensive weight removal while preserving accuracy, through iterative pruning and reintroduction phases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weights are removed from neural network to reduce memory footprint, then model efficiency is improved, but accuracy is significantly lost
Solution Approach 1:
The method performs preliminary training of a dense teacher network before using it to distill knowledge into the sparse student network. This preliminary action ensures that the teacher network has full accuracy capabilities before the pruning process begins, allowing it to guide the student network's training and prevent accuracy loss during weight removal.
Solution Approach 2:
The teacher network serves as an intermediary that transfers knowledge to the student network during the distillation process. Instead of training the student network directly on raw data, it learns from the teacher's softened predictions and internal representations, enabling the student to achieve good accuracy with fewer weights.
2Productivity
If more weights are removed from neural network, then model efficiency is improved, but the number of removable weights is limited before performance impact
Solution Approach 1:
The distillation process incorporates feedback mechanisms where the student network's predictions are continuously compared with the teacher network's softened predictions during training. This feedback loop allows the student to learn from its errors relative to the teacher, enabling effective learning even with significantly pruned architecture.
Solution Approach 2:
The method changes the temperature parameter in the softmax function to produce softened predictions from the teacher network. This parameter change allows the student network to learn from a broader distribution of class probabilities rather than just the hard labels, improving its ability to generalize with fewer weights.
3Reliability
If knowledge distillation is used to train sparse model, then accuracy preservation is improved, but training complexity increases
Solution Approach 1:
The method merges multiple training objectives into a single distillation loss function that combines knowledge transfer from the teacher network with standard classification accuracy optimization. This unified approach simplifies the training process compared to separate training stages while maintaining accuracy preservation.
Data Source
AI summary
Training a neural network (NN) may include training a NN N, and for S, a version of N to be sparsified (e.g. a copy of N), removing NN elements from S to create a sparsified version of S, and training S using outputs from N (e.g. “distillation”). A boosting or reintroduction phase may follow sparsification: training a NN may include for a trained NN N and S, a sparsified version of N, re-introducing NN elements previously removed from S, and training S using outputs from N. The boosting phase need not use a NN sparsified by “distillation.” Training and sparsification, or training and reintroduction, may be performed iteratively or over repetitions.