AI Inference Resource Scheduling for Edge Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying machine learning models on edge devices in IoT systems is challenging due to resource constraints, as they require significant compute resources, especially with deep neural nets and reliance on scarce and expensive GPUs, and existing systems like Kubernetes do not allow efficient sharing or allocation of hardware resources.
Innovation Solution
An AI inference service is used to schedule machine learning models across hardware resources in a clustered computing system by analyzing compute resource metrics and using a scoring algorithm to assign models to suitable resources, ensuring efficient execution based on execution priorities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machine learning models are deployed on edge devices to enable local data processing, then data processing capability is improved, but resource consumption increases significantly
Solution Approach 1:
The patent introduces a centralized analytics system as an intermediary between edge devices and cloud infrastructure. This mediator coordinates ML model deployment, manages hardware accelerator allocation, and balances computation between edge and cloud to optimize resource utilization while maintaining local processing capabilities
Solution Approach 2:
The system segments the computing workload by separating data collection at the edge from intensive ML inference. Edge devices perform lightweight preprocessing while heavy computational tasks are offloaded to centralized hardware accelerators, dividing the resource burden across multiple levels of the system hierarchy
2Speed
If hardware accelerators are used to execute ML models on edge devices, then execution speed is improved, but hardware complexity and cost increase
Solution Approach 1:
The centralized analytics system creates a pool of hardware accelerators that can be dynamically allocated to serve multiple edge devices and various ML workloads. This universal resource pool eliminates the need for each edge device to have dedicated specialized hardware, reducing individual device complexity while maintaining high execution speeds through shared acceleration resources
Solution Approach 2:
The centralized analytics system acts as an intermediary that manages the complexity of hardware accelerator allocation. It handles the intricate task of matching ML models with suitable hardware resources, shielding edge devices from hardware complexity while enabling fast execution through coordinated resource allocation
3Adaptability or versatility
If ML models are deployed across diverse edge devices with varying hardware capabilities, then system adaptability is improved, but scheduling complexity increases
Solution Approach 1:
The centralized analytics system serves as an intermediary that abstracts the diversity of edge device hardware capabilities. It maintains knowledge of various hardware configurations and automatically matches ML models with compatible devices, enabling broad adaptability while keeping the scheduling logic centralized and manageable rather than distributed and complex across each device
Solution Approach 2:
The system manages hardware diversity by parameterizing device capabilities and model requirements. The centralized scheduler adjusts allocation decisions based on varying hardware parameters (processor type, memory, accelerator availability), systematically handling adaptability through parameter-based matching rather than complex conditional logic at each edge device
Data Source
AI summary
Systems and methods described herein generally relate to compute node resource scheduling. AI inference services described herein may receive a request to execute a machine learning model in a clustered edge system. To determine which hardware resource comprising computing nodes of the clustered edge system on which to execute the machine learning model, AI inference services may compare the computational workload of the machine learning model, with the computational abilities and functions of the hardware resources. In examples, the comparison is based on a scheduling algorithm, including an identification stage to identify candidate hardware resources capable of executing the machine learning model, and a scoring stage to select the best candidate hardware resource for executing the machine learning model. A scheduler may assign the machine learning model to the selected hardware resource for execution by the AI inference services.