Domain-Specific Neural Network Pruning for Edge Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models perform poorly in specific domains due to lack of training data and are inefficiently large, requiring significant resources for computation and storage, making them impractical for deployment in new domains with limited resources.
Innovation Solution
A domain-specific neural network pruning method that trains a backbone network on general datasets and extends it with domain-specific branches using small datasets, followed by iterative pruning to achieve optimal sparsity and accuracy, resulting in a lightweight model suitable for inference in specific domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a full-size neural network is used to maintain high accuracy, then the model performance is improved, but the computing capabilities and storage resources required increase significantly
Solution Approach 1:
The patent extracts and removes redundant neurons and connections from the neural network through pruning operations. Less important neurons are eliminated while preserving the core functional structure, reducing computing resources while maintaining accuracy on target domains.
Solution Approach 2:
The patent applies domain-specific pruning strategies that treat different parts of the network differently. Important neurons for specific domains are preserved while redundant ones are removed, creating locally optimized network structures tailored to each domain's requirements.
2Reliability
If a full-size neural network is used to maintain high accuracy, then the model performance is improved, but the storage resources required increase significantly
Solution Approach 1:
The patent extracts and removes redundant neurons and connections from the neural network through pruning operations. Less important neurons are eliminated while preserving the core functional structure, reducing storage requirements while maintaining accuracy on target domains.
Solution Approach 2:
The patent segments the neural network into domain-specific branches with different sparsity ratios. Each branch is independently pruned according to its domain requirements, allowing selective storage optimization without sacrificing overall model accuracy.
3Reliability
If domain-specific training data is collected to improve performance in specific domains, then the model accuracy is improved, but the time and resources required for data collection and training increase
Solution Approach 1:
The patent performs preliminary training on a large general-domain dataset to establish a robust base model. This pre-trained model serves as a foundation that can be quickly adapted to specific domains with minimal additional training data and time.
Solution Approach 2:
The patent changes the sparsity ratio parameter to adapt the model to different domains. By adjusting this parameter and applying domain-specific pruning, the model can be quickly reconfigured for new domains without extensive retraining.
4Productivity
If the neural network is pruned to reduce size and computing requirements, then the deployment efficiency is improved, but the model accuracy may deteriorate
Solution Approach 1:
The patent applies different sparsity ratios to different domain-specific branches, preserving accuracy-critical neurons in important domains while aggressively pruning redundant neurons in less critical domains. This local optimization maintains overall accuracy while improving deployment efficiency.
Solution Approach 2:
The patent uses feedback from domain-specific performance evaluation to guide the pruning process. Neurons are pruned iteratively based on their impact on domain-specific accuracy, ensuring that pruning does not deteriorate performance on target domains.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for domain-specific pruning of neural networks are described. An exemplary method includes obtaining a first neural network trained based on a first training dataset; obtaining one or more second training datasets respectively from one or more domains; and training, based on the first neural network and the one or more second training datasets, a second neural network comprising the first neural network and one or more branches extended from the first neural network, wherein the second neural network is applicable for inferencing in the one or more domains, and the training comprises: training the one or more branches based respectively on the one or more second training datasets and an output of the first neural network.