Self-tuning DNN Compression via Sparsity Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale Deep Neural Networks (DNNs) are not deployable on resource-limited devices like mobile phones due to high computation intensity and memory requirements, and learning from scratch is not feasible with limited data sets, making it difficult for end-users to develop customized models.
Innovation Solution
A self-tuning model compression methodology that involves inter-layer and intra-layer sparsity analysis, low-rank approximation, pruning, and quantization of pre-trained DNN models to generate a reconfigured model suitable for resource-limited devices, allowing for customized feature learning from limited data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large-scale Deep Neural Networks are used to achieve better learning performance, then model accuracy is improved, but computational complexity and memory requirements increase making deployment on resource-limited devices infeasible
Solution Approach 1:
The patent extracts and removes redundant connections and neurons from the pre-trained DNN model through sparsity analysis and pruning techniques. By identifying and eliminating unnecessary computational elements, the model achieves compression while preserving essential functionality, resolving the contradiction between model accuracy and computational complexity.
Solution Approach 2:
Instead of training a small model from scratch on limited data, the patent inverts the approach by starting with a large pre-trained model and compressing it. This inversion allows the model to benefit from the knowledge embedded in large-scale training while adapting to resource-limited deployment environments through compression techniques.
2Measurement precision
If large-scale Deep Neural Networks are used to achieve better learning performance, then model accuracy is improved, but memory requirements increase making deployment on resource-limited devices infeasible
Solution Approach 1:
The patent extracts redundant parameters and connections from the DNN model through sparsity analysis. By removing zero and near-zero weight connections, the model's memory footprint is significantly reduced while maintaining its predictive accuracy, enabling deployment on devices with limited memory resources.
Solution Approach 2:
The patent discards redundant information in the form of zero and near-zero weight connections identified through sparsity analysis. By selectively removing these unnecessary elements while preserving the essential computational pathways, the model achieves memory compression without sacrificing its core learning capabilities.
3Adaptability or versatility
If end-users attempt to develop customized deep learning models with limited data sets, then model customization is achieved, but learning from scratch is not feasible due to insufficient data
Solution Approach 1:
The patent applies preliminary action by using a pre-trained DNN model that has already learned from large-scale data. This pre-trained model serves as a foundation that can be compressed and adapted to end-user applications with limited data, eliminating the need for training from scratch and enabling feasible customization with insufficient data.
Solution Approach 2:
The patent uses a pre-trained DNN model as a template or copy that can be replicated and compressed for end-user deployment. By copying the structure and learned parameters from a large-scale model and then compressing it, the system enables customization on resource-limited devices without requiring extensive training data.
Data Source
AI summary
A self-tuning model compression methodology for reconfiguring a Deep Neural Network (DNN) includes: receiving a pre-trained DNN model and a data set; performing an inter-layer sparsity analysis to generate a first sparsity result; and performing an intra-layer sparsity analysis to generate a second sparsity result, including: defining a plurality of sparsity metrics for the network; performing forward and backward passes to collect data corresponding to the sparsity metrics; using the collected data to calculate values for the defined sparsity metrics; and visualizing the calculated values using at least a histogram. The methodology further includes: according to the first and second sparsity results, performing low-rank approximation on the pre-trained DNN; pruning the represented DNN model according to the first and second sparsity results; performing quantization on the pruned DNN model according to the first and second sparsity results; and executing the reconfigured model on a user terminal for an end-user application.