Neural Network-Guided Deep Learning Model Compression for Edge Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models are too large and resource-intensive for deployment on devices with limited memory and processing power, leading to extended inference times and high energy consumption, making them impractical for real-time applications.
Innovation Solution
A method and system that automatically compress deep learning models using a neural network trained on a concatenated vector of compression feature, model feature, and dataset feature vectors, selecting an optimal compression strategy based on domain knowledge and performance parameters to maintain accuracy while reducing model size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If deep learning models are deployed on edge devices, then real-time processing capability is improved, but device resource constraints (memory and processing power) are exceeded
Solution Approach 1:
The patent segments the deep learning model into multiple components that can be independently compressed or pruned. Different parts of the model are subjected to different compression techniques based on their importance and impact on performance, allowing the model to be divided into manageable segments that fit within device memory constraints while preserving real-time processing capability.
Solution Approach 2:
The patent changes key parameters of the deep learning model including precision reduction (e.g., from 32-bit floating point to 8-bit integer), layer pruning ratios, and activation function approximations. These parameter changes significantly reduce model size and computational requirements while maintaining acceptable accuracy levels for real-time edge device deployment.
2Measurement precision
If larger deep learning models are used, then model accuracy is improved, but inference time increases
Solution Approach 1:
The patent extracts and removes redundant or less important components from the deep learning model through techniques such as layer pruning, neuron pruning, and connection pruning. By taking out unnecessary elements, the model size and computational complexity are reduced, leading to faster inference times while preserving the essential functionality and accuracy of the model.
Solution Approach 2:
The patent applies partial compression to different parts of the model, using aggressive compression techniques on less critical layers and more conservative techniques on critical layers. This selective approach ensures that the most important model components maintain high accuracy while less critical components are compressed to reduce overall inference time.
3Reliability
If larger deep learning models are deployed, then model performance is improved, but energy consumption increases
Solution Approach 1:
The patent changes precision parameters from high-precision 32-bit floating point to lower-precision 8-bit or 16-bit formats, which significantly reduces the computational energy required for matrix multiplications and other operations. This parameter change maintains acceptable model performance while reducing energy consumption to levels suitable for battery-powered edge devices.
Solution Approach 2:
The patent extracts and removes computationally expensive operations and layers from the model that consume disproportionate amounts of energy. By eliminating redundant computations and focusing only on essential processing steps, the model achieves acceptable performance with significantly reduced energy consumption for deployment on resource-constrained devices.
4Reliability
If heavy deep learning models are transferred over network, then model capability is improved, but network bandwidth requirements increase
Solution Approach 1:
The patent extracts only the essential model parameters and weights required for deployment, removing redundant data and compressing the model representation. This extraction process reduces the data transmission volume from gigabytes to manageable sizes that can be efficiently transferred over networks with limited bandwidth while preserving the core model capability.
Solution Approach 2:
The patent changes the parameter representation from high-precision 32-bit floating point format to compressed 8-bit or 16-bit formats, reducing the data size by a factor of 4x or 8x respectively. This parameter compression enables efficient network transfer of model updates and deployments without significant loss of model capability.
Data Source
AI summary
A method and system of compressing a first deep learning (DL) model is disclosed. A processor receives a verified DL model. The verified DL model is converted into a standard DL model based on a framework corresponding to a plurality of provisional compression types. A compression strategy is selected from a plurality of compression strategies using a neural network (NN) based on determining a compression feature vector based on a knowledge graph. A concatenated vector is determined based on a model feature vector, a dataset feature vector and compression feature vector. The NN is trained based on the concatenated vector. A bias of the NN is trained based on a model score corresponding to the standard NN. A compression embedding is determined corresponding to the selected compression strategy.