Neural Network Guided Deep Learning Model Compression for Edge Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models are too large and resource-intensive for deployment on devices with limited memory and processing power, leading to extended inference times and high energy consumption, which hinders their use in real-time applications and data transfer over limited bandwidth.
Innovation Solution
A method and system that automatically compress deep learning models using a neural network trained on a concatenated vector of compression feature, model feature, and dataset feature vectors, selecting an optimal compression strategy based on domain knowledge and performance parameters to maintain accuracy while reducing model size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If deep learning models are deployed on resource-constrained devices, then real-time processing capabilities are achieved, but model size and computational requirements exceed device limitations
Solution Approach 1:
The patent segments the deep learning model into multiple components that can be independently compressed or approximated. Different compression techniques are applied to different parts of the model based on their importance and characteristics, allowing selective reduction of model size while preserving critical functionality for real-time processing.
Solution Approach 2:
The patent changes key parameters of the deep learning model including precision reduction (e.g., from 32-bit to 8-bit integers), sparsity induction, and architectural modifications. These parameter changes enable significant model size reduction while maintaining acceptable accuracy for real-time deployment on edge devices.
2Reliability
If larger deep learning models are used, then model accuracy is improved, but inference time and energy consumption increase
Solution Approach 1:
The patent applies partial compression techniques where only certain layers or filters of the neural network are compressed beyond what would be minimally required. This selective over-compression of specific components allows achieving lower inference times while maintaining overall model accuracy through strategic preservation of critical pathways.
Solution Approach 2:
The patent introduces dynamic elements into the compressed model architecture, such as dynamic quantization that adapts precision requirements based on input characteristics, and dynamic pruning that activates different model pathways depending on the specific inference task, thereby optimizing the balance between accuracy and speed for each individual case.
3Power
If larger deep learning models are deployed, then processing capability is improved, but energy consumption and battery depletion increase
Solution Approach 1:
The patent employs low-precision numerical representations (such as 8-bit or 4-bit integers instead of 32-bit floats) that require significantly less computational energy to process. This substitution of high-precision operations with lower-precision equivalents reduces the energy cost of each inference operation, enabling sustained operation on battery-powered devices.
Solution Approach 2:
The patent extracts and removes computationally intensive components from the model that contribute minimally to the final output. By identifying and eliminating redundant calculations, heavy matrix operations, and non-critical layers, the remaining model structure requires substantially less processing power and energy consumption while retaining core functionality.
4Quantity of substance
If deep learning models are compressed, then model size is reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent implements feedback mechanisms during the compression process where the model's performance on validation data is continuously monitored. Based on this feedback, compression parameters such as pruning thresholds, quantization levels, and distillation settings are iteratively adjusted to achieve the maximum possible compression while maintaining accuracy within acceptable boundaries.
Solution Approach 2:
The patent introduces intermediary components such as knowledge distillation teachers that guide the compression process. A larger, more accurate teacher model transfers its knowledge to the compressed student model through carefully designed loss functions and training procedures, enabling the student model to achieve high accuracy despite its reduced size.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (600) and system (100) of compressing a first deep learning (DL) model is disclosed. A processor receives a verified DL model. The verified DL model is converted into a standard DL model based on a framework corresponding to a plurality of provisional compression types. A compression strategy is selected from a plurality of compression strategies using a neural network (NN) based on determining a compression feature vector based on a knowledge graph. A concatenated vector is determined based on a model feature vector, a dataset feature vector and compression feature vector. The NN is trained based on the concatenated vector. A bias of the NN is trained based on a model score corresponding to the standard NN. A compression embedding is determined corresponding to the selected compression strategy.