Sparse Optimizer Pruning Neural Network Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) for computer vision tasks are resource-intensive, making them impractical for deployment on resource-constrained devices like mobile phones, smart sensors, and autonomous drones or robots due to high storage and computation costs.
Innovation Solution
A sparse optimizer system is implemented to prune neural networks, specifically deep generative models (DGMs), by reducing the number of non-zero parameters, thereby compressing the model size and allowing it to be executed on resource-scarce devices without modifying the trained model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used for computer vision tasks, then accuracy is improved, but storage and computation costs increase
Solution Approach 1:
The patent extracts and removes redundant parameters from the neural network model through pruning techniques. By identifying and eliminating unnecessary connections and neurons, the model maintains its accuracy while significantly reducing the number of parameters that require storage, thus resolving the contradiction between accuracy and storage cost.
Solution Approach 2:
The patent changes the parameter density of the neural network by transforming it from a dense structure to a sparse structure. This parameter transformation reduces the quantity of parameters needed while preserving the essential functional parameters that contribute to accuracy, thereby reducing storage requirements without sacrificing performance.
2Measurement precision
If deep neural networks are used for computer vision tasks, then accuracy is improved, but computation cost increases
Solution Approach 1:
The patent extracts and eliminates redundant computational elements from the neural network through pruning. By removing unnecessary parameters and connections, the computation required for inference is significantly reduced while the remaining essential parameters maintain the model's accuracy, thus resolving the contradiction between accuracy and computation cost.
Solution Approach 2:
The patent transforms the network architecture from dense to sparse by changing the parameter distribution. This parameter change reduces the number of floating-point operations required during inference while preserving the critical parameters that determine accuracy, thereby reducing computation cost without compromising performance.
3Volume of stationary object
If model size is reduced through pruning, then deployment on edge devices is enabled, but model complexity increases
Solution Approach 1:
The patent applies pruning techniques as a preliminary action during the model preparation phase, before deployment on edge devices. By pre-identifying and removing redundant parameters, the model is simplified in advance, reducing both its size and complexity. This preliminary simplification makes the model suitable for edge device deployment without requiring complex runtime optimizations.
Data Source
AI summary
A neural network pruning system can sparsely prune neural network models using an optimizer based approach that is agnostic to the model architecture being pruned. The neural network pruning system can prune by operating on the parameter vector of the full model and the gradient vector of the loss function with respect to the model parameters. The neural network pruning system can iteratively update parameters based on the gradients, while zeroing out as many parameters as possible based a preconfigured penalty.