Multi-Model Edge Scheduling Across CPU, GPU, and VPU
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern deep neural networks (DNNs) face challenges in deployment on single edge devices due to their high computational demands, leading to the need for multiple heterogeneous edge devices, which complicates scheduling and increases latency.
Innovation Solution
A hardware adaptive multi-model scheduling framework that iteratively allocates neural network models to edge devices based on performance metrics, using model first scheduling (MFS) and hardware first scheduling (HFS) methods to optimize deployment on heterogeneous platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If multiple heterogeneous edge devices are used to deploy deep neural networks, then computational capacity is sufficient to handle complex models, but scheduling complexity and latency increase
Solution Approach 1:
The patent transforms the scheduling problem from a complex many-to-many matching problem into a simpler assignment problem by changing the selection parameter. Instead of considering all possible model-device combinations simultaneously, the system iteratively selects devices based on worst-performance parameters (latency, energy consumption) and assigns models sequentially, reducing the computational complexity of the scheduling decision process while maintaining optimal resource utilization across heterogeneous edge devices
Solution Approach 2:
The patent segments the scheduling process into distinct phases: performance evaluation, worst-case identification, device selection, and model assignment. By dividing the complex scheduling task into these manageable segments executed iteratively, the system reduces overall scheduling complexity while ensuring that each segment contributes to optimizing the final deployment configuration across multiple heterogeneous edge devices
2Measurement precision
If deeper neural network layers are added to improve accuracy, then model performance increases, but computational requirements and deployment difficulty increase
Solution Approach 1:
The patent applies local quality by matching specific deep neural network models with edge devices that have locally optimized characteristics for those models. Instead of requiring all devices to have uniform high computational power, the system identifies and assigns models to devices with specific local strengths (e.g., GPU-accelerated devices for computationally intensive layers, devices with optimized neural network libraries), enabling deployment of deep accurate models across heterogeneous hardware without requiring every device to be universally capable
Solution Approach 2:
The patent introduces a new dimension of optimization by considering multiple performance metrics simultaneously (latency, energy consumption, accuracy) rather than focusing solely on computational speed. This multi-dimensional approach allows the system to deploy accurate deep neural networks by selecting devices that optimize the overall performance profile, not just raw computational power, thereby reducing deployment complexity while maintaining high model accuracy
3Productivity
If iterative allocation processes are used to optimize model-device matching, then deployment efficiency improves, but computation time for scheduling increases
Solution Approach 1:
The patent applies preliminary action by pre-evaluating and storing performance metrics for each edge device before the actual scheduling decision. The system collects latency, energy consumption, and other performance data in advance, creating a performance profile for each device. This preliminary characterization allows the iterative allocation process to make informed decisions quickly without performing complex real-time analysis, thus improving deployment efficiency while minimizing additional scheduling computation time
Data Source
AI summary
Modem deep neural network (DNN) models have many layers with a single layer potentially involving large matrix multiplications. Such heavy calculation brings challenges to deploy such DNN models on a single edge device, which has relatively limited computation resources. Therefore, multiple and even heterogeneous edge devices may be required for applications with stringent latency requirements. Disclosed in the present patent documents are embodiments of a model scheduling framework that schedules multiple models on a heterogeneous platform. Two different approaches, model first scheduling (MFS) and hardware first scheduling (HFS), are presented to allocate a group of models for a service into corresponding heterogeneous edge devices, including CPU, VPU and GPU. Experimental results prove the effectiveness of the MFS and HFS methods for improving the inference speed of single and multiple AI-based services.