Embedded Model Compression for Hardware-Aware Inference Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional model compression techniques primarily performed in server environments fail to consider the differences between server and embedded environments, leading to insufficient acceleration effects and suboptimal performance in resource-constrained embedded devices.
Innovation Solution
A method and device for model compression that adapts to resource-constrained embedded environments by using a model converter to determine computable layers, applying compression techniques, and evaluating performance using actual data, with features like pruning and depth compression, adaptive batch normalization, and sparse updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If model compression is performed in server environment using conventional techniques, then model size is reduced, but inference acceleration effect is insufficient in embedded environment
Solution Approach 1:
The patent changes the compression parameters and techniques based on the target embedded device's hardware capabilities. By analyzing the device's computational resources, memory, and power constraints, the system adjusts compression ratios, quantization levels, and pruning thresholds to optimize both model size reduction and inference speed for the specific embedded environment, rather than using fixed server-oriented compression parameters.
Solution Approach 2:
The patent applies different compression techniques to different parts of the model based on their importance and the embedded device's capabilities. Critical layers maintain higher precision while less critical layers undergo aggressive compression. This localized quality adjustment ensures that the compressed model achieves size reduction while preserving essential inference performance on resource-constrained devices.
2Device complexity
If model compression is performed in server environment, then compression process is simplified, but model performance does not translate to actual service environment
Solution Approach 1:
The patent performs preliminary analysis of the target embedded device's hardware characteristics before compression. By pre-characterizing the device's computational capabilities, memory constraints, and power limitations, the system prepares device-specific compression parameters and techniques in advance, ensuring that the compressed model is optimized for the actual deployment environment rather than assuming server-like conditions.
Solution Approach 2:
The patent implements feedback mechanisms where the compressed model's performance is evaluated on the actual embedded device, and compression parameters are iteratively adjusted based on measured inference speed, accuracy, and resource consumption. This closed-loop approach ensures that the final compressed model achieves both size reduction and performance consistency in the target embedded environment.
3Adaptability or versatility
If deep learning model is operated in embedded device, then service deployment is expanded, but computational resources are insufficient
Solution Approach 1:
The patent segments the deep learning model into multiple components or stages that can be selectively executed based on the embedded device's available computational resources. By dividing the model into essential and optional parts, the system can deploy the minimum required functionality on resource-constrained devices while maintaining the option to include additional features when resources permit, thus expanding deployment versatility across diverse hardware platforms.
Data Source
AI summary
In an embodiment a method includes receiving a deep learning model loaded on a target device and a computation list supported by a model converter for the target device, determining whether a computation for each layer of the deep learning model is able to be accelerated on the target device based at least in part on the computation list, acquiring a compressed model by maintaining a computation corresponding to one layer as it is, and applying a compression technique to the deep learning model when the one layer of the deep learning model is determined to include a computation unable to be accelerated on the target device, and acquiring the compressed model by changing the computation corresponding to the one layer to another computation, and then applying the compression technique to the deep learning model when the one layer is determined to include the computation able to be accelerated on the target device.