DL Model Partitioning With Compression and Quantization for Edge Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Porting and optimizing deep learning models on low-resource embedded targets in devices like Industrial Internet of Things (IIoT), Robotics, and Cyber-physical systems poses challenges due to large model sizes, which can prevent loading at runtime and affect accuracy with arbitrary compression and quantization.
Innovation Solution
A method involving partitioning DL models into layers, applying compression and quantization configurations to achieve maximum inference accuracy, minimum latency, and low power consumption across multiple processing elements, with dynamic adjustments for different hardware types, including COTS accelerators, using techniques like quantization aware retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the DL model size is reduced through quantization and compression, then the model can be loaded and processed on low-resource hardware, but the accuracy represented by the model deteriorates
Solution Approach 1:
The patent segments the DL model into multiple partitions that can be distributed across different processing elements. Each partition can be independently optimized for accuracy while the collective system achieves the desired balance between model size and accuracy. This segmentation allows selective compression in different model regions without uniformly degrading accuracy.
Solution Approach 2:
Different partitions of the DL model are assigned different compression and quantization configurations based on their specific requirements. Critical layers maintain higher accuracy while less critical layers undergo more aggressive compression. This local quality approach enables optimized accuracy-compression tradeoff for each model region rather than uniform treatment.
2Speed
If the DL model is partitioned across multiple processing elements, then the inference latency is reduced, but the device complexity increases
Solution Approach 1:
The processing elements are designed with multi-functionality to handle different types of computational operations. The same PE can execute different layers or partitions of the model, and the system can dynamically assign workloads. This universality reduces the need for specialized hardware for each function, thereby reducing overall system complexity while maintaining low latency.
Solution Approach 2:
The system employs dynamic workload distribution where the assignment of model partitions to processing elements can change based on real-time conditions. This dynamic approach allows the system to optimize latency by routing computations to available PEs while adapting to varying hardware capabilities and conditions, reducing the need for rigid complex hardware designs.
3Use of energy by moving object
If compression configurations are applied to reduce model size, then the power consumption is reduced, but the inference accuracy deteriorates
Solution Approach 1:
Instead of uniformly applying maximum compression to the entire model, the system applies partial compression selectively to specific partitions or layers. This allows the system to achieve significant power savings through compression in less critical regions while maintaining sufficient accuracy in regions where it matters most, optimizing the energy-accuracy tradeoff.
Solution Approach 2:
The system changes compression parameters (such as quantization bit-width, compression ratio) dynamically based on the specific requirements of each model partition. By adjusting these parameters rather than using fixed compression levels, the system can optimize power consumption for each region while maintaining accuracy thresholds, achieving better overall energy efficiency.
Data Source
AI summary
Small and compact Deep Learning models are required for embedded AI in several domains. In many industrial use-cases, there are requirements to transform already trained models to ensemble embedded systems or re-train those for a given deployment scenario, with limited data for transfer learning. Moreover, the hardware platforms used in embedded application include FPGAs, AI hardware accelerators, System-on-Chips and on-premises computing elements (Fog/Network Edge). These are interconnected through heterogenous bus/network with different capacities. Method of the present disclosure finds how to automatically partition a given DNN into ensemble devices, considering the effect of accuracy—latency power—tradeoff, due to intermediate compression and effect of quantization due to conversion to AI accelerator SDKs. Method of the present disclosure is an iterative approach to obtain a set of partitions by repeatedly refining the partitions and generating a cascaded model for inference and training on ensemble hardware.