Self-tuning Neural Network Compression via Sparsity and Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale deep neural networks are challenging to deploy on resource-limited devices due to high computation intensity and memory requirements, making it difficult to perform inference on mobile phones and embedded devices, and learning from scratch is not feasible with limited data sets.
Innovation Solution
A method for compressing pre-trained deep neural networks by increasing sparsity through node removal, eliminating batch normalization layers, and quantizing weights into fixed-point representation, with iterative adjustments to hyperparameters based on end-user performance metrics to ensure accuracy meets or exceeds target levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large-scale deep neural network is used to achieve high performance on cutting-edge tasks, then accuracy and performance are improved, but computation intensity and memory requirements increase, making deployment on resource-limited devices difficult
Solution Approach 1:
The patent segments the deep neural network into multiple layers and applies different compression strategies to different segments. It removes batch normalization layers from certain segments and applies pruning to weight matrices in other segments, allowing selective compression that maintains accuracy while reducing overall computation intensity for deployment on resource-limited devices
Solution Approach 2:
The patent changes the parameter representation from floating-point to fixed-point format, reducing the memory footprint and computation requirements. It also modifies the network architecture parameters by removing batch normalization layers and applying weight pruning, thereby reducing computation intensity while maintaining model accuracy for edge device deployment
2Reliability
If a large-scale deep neural network is used to achieve high performance, then model capability is improved, but memory storage requirements increase, making it unsuitable for resource-limited end-user devices
Solution Approach 1:
The patent changes the data type of weight parameters from floating-point to fixed-point representation, significantly reducing the memory storage requirements. This parameter change allows the model to be deployed on resource-limited end-user devices with smaller memory capacities while maintaining acceptable accuracy performance
Solution Approach 2:
The patent extracts and removes batch normalization layers from the neural network architecture, reducing the overall model size and memory requirements. This extraction is performed selectively to maintain the essential computational functionality while reducing memory storage demands for deployment on devices with limited memory capacity
3Device complexity
If compression is applied to reduce model size and computation intensity, then deployment feasibility is improved, but model accuracy may deteriorate
Solution Approach 1:
The patent applies partial compression by selectively removing only certain batch normalization layers and applying weight pruning with specific thresholds rather than complete compression. This partial action reduces computation intensity while preserving sufficient accuracy for the target application domain
Solution Approach 2:
The patent implements a feedback mechanism that evaluates model accuracy after compression and adjusts the compression parameters accordingly. If accuracy deteriorates beyond acceptable thresholds, the compression level is reduced or different compression strategies are applied, ensuring that the final model maintains required accuracy while achieving desired computational efficiency
4Reliability
If iterative compression with hyper parameter adjustment is performed, then accuracy performance is optimized, but compression time and computational cost increase
Solution Approach 1:
The patent performs a limited number of iterative compression steps with hyper parameter adjustment rather than exhaustive optimization. This partial action achieves sufficient accuracy optimization for practical applications while significantly reducing the time and computational cost associated with complete iterative optimization
Data Source
AI summary
A method of compressing a pre-trained deep neural network model includes inputting the pre-trained deep neural network model as a candidate model. The candidate model is compressed by increasing sparsity of the candidate, removing at least one batch normalization layer present in the candidate model, and quantizing all remaining weights into fixed-point representation to form a compressed model. Accuracy of the compressed model is then determined utilizing an end-user training and validation data set. Compression of the candidate model is repeated when the accuracy improves. Hyper parameters for compressing the candidate model are adjusted, then compression of the candidate model is repeated when the accuracy declines. The compressed model is output for inference utilization when the accuracy meets or exceeds the end-user performance metric and target.