Edge–Cloud Deep Neural Network Inference With Dynamic Split Points
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge devices face challenges in performing real-time inference of complex deep neural networks due to resource constraints, and transmitting images to cloud servers for processing is time-consuming due to wireless network limitations.
Innovation Solution
A deep neural network-based system that splits inference tasks between edge devices and cloud servers, using quantized features at specific layers, with dynamic adjustment of split points based on resource analysis, and includes a feature distribution matching unit for precision restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If edge devices transmit images to cloud servers for processing, then inference accuracy is maintained, but transmission time increases due to wireless network limitations
Solution Approach 1:
The inference process is segmented into two parts: feature extraction is performed on the edge device, and only the extracted features (not the full image) are transmitted to the cloud server for final processing. This segmentation reduces transmission data volume while maintaining inference accuracy.
Solution Approach 2:
The patent extracts only the essential feature representations from images at the edge device before transmission. By taking out only the necessary feature data rather than transmitting complete images, the system reduces transmission time while preserving the information needed for accurate inference.
2Loss of substance
If quantization is applied to reduce transmission features to low bits, then transmission overhead is reduced, but model operation becomes restricted to specific split layers
Solution Approach 1:
The system dynamically determines the optimal split point for quantization based on resource conditions. The quantization module can be adaptively positioned at different layers of the neural network, allowing the system to adjust between transmission reduction and operational flexibility based on current resource availability.
Solution Approach 2:
The patent changes the parameter of quantization bit-depth dynamically. By adjusting the quantization level and position based on resource conditions, the system optimizes transmission overhead while maintaining sufficient model operation flexibility for different deployment scenarios.
3Productivity
If independent models are constructed for each split point, then optimal performance is achieved for specific conditions, but system complexity increases
Solution Approach 1:
The patent implements a universal model architecture that can function across multiple split points. Instead of constructing separate independent models for each split point, a single model design handles variable quantization positions, reducing system complexity while maintaining optimal performance through adaptive configuration.
Solution Approach 2:
The system uses dynamic configuration to adjust the quantization split point based on resource conditions rather than requiring separate static models for each scenario. This dynamic approach allows one model to adapt to different performance requirements, reducing overall system complexity.
Data Source
AI summary
The present disclosure relates to a deep neural network-based real-time inference apparatus, system, and method, and more particularly, to a deep neural network-based real-time inference apparatus, system, and method capable of accelerating image inference.