Knowledge-Preserving Neural Network Pruning via Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network pruning techniques often destroy the knowledge learned by fine-tuned models when reducing the size of neural networks, making them inefficient for deployment on devices with limited resources.
Innovation Solution
A knowledge-preserving neural network pruning process that involves constructing a student network from a pre-trained teacher network through simultaneous knowledge distillation and network pruning, using a distillation loss function to minimize output differences and sparse pruning to reduce neuron activations, thereby preserving task-specific and general-purpose knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If existing neural network pruning techniques are applied to reduce the size of a fine-tuned neural network, then the network size is reduced, but the knowledge learned by the model is destroyed
Solution Approach 1:
The patent applies preliminary action by performing knowledge distillation before pruning. The student network is first trained to replicate the teacher network's knowledge through knowledge distillation, ensuring knowledge preservation. Only after this preliminary knowledge transfer is the pruning applied, thus preventing knowledge loss that would occur if pruning were applied directly to the fine-tuned model.
Solution Approach 2:
The patent introduces an intermediary mechanism - the student network - that acts as a mediator between the teacher network and the pruned model. The student network receives knowledge from the teacher network through knowledge distillation and then undergoes pruning, serving as an intermediate structure that preserves knowledge while enabling size reduction.
2Adaptability or versatility
If a fine-tuned neural network is deployed on devices with limited computing capabilities and storage resources, then the model can serve the task, but the device cannot host the full-size model
Solution Approach 1:
The patent applies parameter changes by transforming the model through knowledge distillation and pruning operations. The student network is trained with adjusted parameters to match the teacher network's behavior, and then pruning modifies the parameter structure by setting certain weights to zero, thereby reducing model size while maintaining functional parameters that preserve knowledge.
Solution Approach 2:
The patent extracts unnecessary components from the neural network through pruning. By identifying and removing redundant neurons and connections while keeping essential knowledge-preserving parameters, the model size is reduced to fit resource-constrained devices while maintaining the extracted knowledge from the teacher network.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for knowledge-preserving sparse pruning on neural networks are described. An exemplary method includes obtaining a pre-trained machine learning model trained based on a plurality of general-purpose training data; training a task-specific machine learning model by tuning the pre-trained machine learning model based on a plurality of task-specific training data corresponding to a task; constructing a student network based on the task-specific machine learning model; simultaneously performing (1) knowledge distillation from the trained task-specific machine learning model as a teacher network to the student network and (2) network pruning on the student network; and obtaining the trained student network for serving the task.