ML Model Scheduling on Heterogeneous Edge Resources
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment of deep neural network models on edge devices with limited computation resources is challenged by their high complexity, leading to increased inference times and latency, especially when multiple models are executed on heterogeneous devices with varying hardware configurations.
Innovation Solution
A resource mapping system that adapts to machine learning models, using either a model-first or hardware-first scheduling scheme to allocate processing resources efficiently, determining optimal mappings based on timing matrices and executing models on suitable resources to minimize latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural network models with high complexity are deployed on edge devices with limited computation resources, then model accuracy is improved, but inference latency increases
Solution Approach 1:
The patent segments the deployment problem by separating model selection from resource allocation. It divides edge devices into different capability tiers and assigns different model complexities to different devices based on their computational resources, thereby balancing accuracy requirements with latency constraints across the heterogeneous edge device ecosystem.
Solution Approach 2:
The patent changes the parameter of model complexity to match device capabilities. By adjusting model depth, width, and computational requirements based on the specific edge device's processing power, memory, and energy constraints, the system optimizes the trade-off between achieving high accuracy and maintaining low inference latency for each device.
2Adaptability or versatility
If multiple ML models are executed on heterogeneous edge devices, then service capabilities are improved, but resource allocation complexity increases
Solution Approach 1:
The patent creates a universal scheduling framework that can handle multiple ML models across heterogeneous edge devices through a common resource allocation mechanism. The system uses standardized device profiling and model characterization to enable a single scheduling algorithm to manage diverse workloads, reducing the complexity of allocating resources for multiple services on varied hardware platforms.
Solution Approach 2:
The patent implements dynamic resource allocation that adapts to changing workload requirements and device states. The scheduling system continuously monitors device performance, model execution patterns, and resource utilization to dynamically adjust model placement and allocation strategies, enabling flexible service capabilities while managing allocation complexity through real-time adaptation rather than static configurations.
3Productivity
If model complexity is increased to improve accuracy, then performance on benchmark tasks is improved, but deployment feasibility on resource-constrained devices deteriorates
Solution Approach 1:
The patent applies local quality by matching model characteristics to specific device capabilities. Instead of using a uniform approach, the system profiles each edge device's computational resources, memory, and energy constraints, then selects and configures models with appropriate complexity levels for each device's specific capabilities, ensuring high accuracy where resources permit while maintaining deployment feasibility on constrained devices.
Data Source
AI summary
A system determines a timing matrix corresponding to inference times taken for a number of machine learning (ML) models to be executed by a number of processing resources of a computing device. The processing resources includes at least a first and a second type of processing resources. The system applies a service-specific model-first scheduling scheme or a service-specific hardware-first scheduling scheme to obtain corresponding service-specific mappings. The system determines a best mapping from the corresponding service-specific mappings. The system schedules each of the ML models to a corresponding processing resource from the processing resources according to the best mapping. The system executes the ML models using corresponding mapped processing resources.