Edge-Cloud Inference Routing via Confidence Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The deployment of machine learning models on edge devices is challenging due to limited computing resources and constraints such as size, power, and weight, especially for models like deep neural networks that require significant resources.
Innovation Solution
A computer-implemented method and system that control an edge device by receiving sensor data, generating local prediction data using a local machine learning model, and determining whether to transmit a query with the sensor data to a cloud computing system based on confidence score data and query threshold data. The method assigns local or cloud prediction data as the prediction result depending on the assessment and uses this result to control the edge device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are deployed on edge devices, then local prediction capability is improved, but computing resource constraints are violated
Solution Approach 1:
The system segments the machine learning inference process between edge devices and cloud computing systems. Edge devices perform local predictions using lightweight models for immediate responses, while complex queries are offloaded to the cloud for sophisticated processing. This segmentation allows each component to operate within its resource constraints while contributing to the overall system capability.
Solution Approach 2:
The system introduces an intermediary assessment mechanism that evaluates whether to transmit queries to the cloud based on confidence scores and query thresholds. This intermediary layer decides the split between local and cloud processing, enabling edge devices to leverage cloud resources when needed while maintaining autonomous operation for simple tasks.
2Measurement precision
If queries are transmitted to cloud computing system, then prediction accuracy is improved, but network bandwidth and response time are worsened
Solution Approach 1:
The system implements feedback through confidence score generation and query threshold comparison. The edge device assesses its own prediction confidence and uses this feedback to determine whether to offload to the cloud. This feedback mechanism optimizes the balance between accuracy and response time by avoiding unnecessary cloud queries for high-confidence local predictions.
Solution Approach 2:
The system dynamically adjusts the query transmission decision based on real-time conditions including confidence scores, query thresholds, and network status. This dynamic approach allows the system to adapt to varying workloads and network conditions, transmitting queries to the cloud only when accuracy benefits outweigh the time cost.
3Loss of time
If local machine learning model is used, then response time is improved, but prediction accuracy is worsened
Solution Approach 1:
The system changes the parameter of model complexity based on task requirements. For time-critical tasks with acceptable accuracy thresholds, the system uses lightweight local models. For tasks requiring high accuracy, the system switches to cloud-based sophisticated models. This parameter change strategy optimizes the trade-off between response time and accuracy.
Data Source
AI summary
A computer-implemented system and method relate an edge device with a local machine learning model, which generates local prediction data and confidence score data, in response to sensor data. Query threshold data is received from a cloud computing system. An assessment result is assessed using the confidence score data and the query threshold data. The assessment result indicates whether or not to generate a query with the sensor data for transmission to the cloud computing system. The local predication data is assigned as a prediction result when the assessment result indicates that the query is not being generated and transmitted. The cloud prediction data is assigned as the prediction result when the assessment result indicates that the query is being generated and transmitted. The cloud prediction data is received from the cloud computing system in response to the query. The edge device is controlled using the prediction result.