Edge-Cloud Inference Routing for Resource-Limited ML Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment of machine learning models on edge devices is challenging due to limited computing resources and constraints, with deep neural networks requiring significantly more resources than what smaller edge devices can offer.
Innovation Solution
A system that intelligently manages machine learning inferences across edge-cloud systems by using lightweight models on edge devices for initial processing and offloading queries to cloud computing systems based on confidence scores, network latency, and system load, leveraging reinforcement learning to optimize resource allocation and query management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If machine learning models are deployed on edge devices, then local processing capability is improved, but computing resource constraints are exceeded
Solution Approach 1:
The system segments the machine learning workload by deploying lightweight models on edge devices for initial processing and offloading complex inference tasks to cloud computing systems. This segmentation allows edge devices to handle simple tasks within their resource constraints while leveraging cloud power for computationally intensive operations.
Solution Approach 2:
The system merges edge computing and cloud computing into a unified edge-cloud architecture. Edge devices and cloud systems work together collaboratively, with edge devices performing local preprocessing and cloud systems providing supplemental computational power, thereby resolving the resource constraint problem.
2Measurement precision
If deep neural networks are used, then model accuracy is improved, but resource requirements exceed edge device capabilities
Solution Approach 1:
The system segments the neural network processing by deploying simplified or pruned versions of deep neural networks on edge devices, while offloading the most computationally intensive layers or complex inference tasks to cloud computing systems with sufficient resources to handle deep neural networks effectively.
Solution Approach 2:
Different computing resources are allocated to different locations based on their capabilities: edge devices use lightweight models suitable for their constrained environment, while cloud systems host comprehensive deep neural networks. Each location uses the appropriate model complexity for its resource profile.
3Loss of time
If all queries are processed locally, then response time is improved, but inference accuracy deteriorates due to model limitations
Solution Approach 1:
The system performs preliminary processing locally on the edge device using lightweight models to handle simple queries quickly. When the lightweight model detects complex queries or low-confidence predictions, it preliminarily prepares the data and offloads it to the cloud system for more accurate processing, thereby maintaining fast response for simple cases while ensuring accuracy for complex cases.
Solution Approach 2:
The system introduces an intermediary mechanism (such as a router or gateway) that mediates between edge devices and cloud systems. This intermediary determines which queries should be processed locally and which should be offloaded to the cloud, optimizing the balance between response time and inference accuracy based on query characteristics and system state.
4Power
If cloud resources are increased, then processing capability is improved, but system cost increases
Solution Approach 1:
The system implements dynamic resource allocation where cloud computing resources are scaled up or down based on real-time demand. During peak periods or when complex queries require additional power, cloud resources are increased. During low-demand periods, resources are reduced to minimize costs. This dynamic adjustment optimizes the balance between processing capability and system cost.
Solution Approach 2:
The system employs feedback mechanisms to monitor system performance, query patterns, and resource utilization. Based on this feedback, the system automatically adjusts the distribution of work between edge and cloud systems, optimizing resource allocation and controlling costs while maintaining required processing capability.
Data Source
AI summary
A system and method relate to managing a cloud computing system. Queries are received from one or more edge devices of a set of edge devices. Each query includes sensor data from the respective edge device. Prediction data is generated via one or more cloud machine learning models, using the sensor data, during a current time period. System state data is generated and indicates a current state of an environment during the current time period. The environment is defined by the cloud computing system and the set of edge devices. A machine learning system generates policy data by optimizing an expected return of a reward with respect to taking a particular action given the system state data. The machine learning system is employed by the cloud computing system. The policy data indicates a recommended action from a set of actions. The cloud computing system performs the recommended action.