Edge Inference Model Distribution for Server Queue Delay Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inference models deployed on central servers often experience delays due to queue buildup when handling high volumes of data, leading to inefficiencies in real-time or near real-time processing, especially in secure dedicated environments where scaling is limited.
Innovation Solution
Inference models are selectively applied on edge devices in response to delays or predicted wait times, allowing edge devices to perform inferences locally when server queues exceed predetermined thresholds, thereby distributing the processing burden and reducing wait times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If inference models are deployed on central servers to handle large volumes of data, then processing capacity is improved, but queue delays increase when server capacity is exceeded
Solution Approach 1:
The patent segments the centralized inference processing into distributed edge inference and selective server inference. Edge devices perform inference locally for time-sensitive data, while the server handles non-urgent data, dividing the processing workload to eliminate queue delays for real-time requirements
Solution Approach 2:
The system performs preliminary actions by deploying trained inference models to edge devices in advance. This allows edge devices to immediately perform inference on incoming data without waiting for server processing, proactively eliminating queue delays before they occur
2Loss of time
If inference models are deployed on edge devices for real-time processing, then response time is improved, but device complexity increases
Solution Approach 1:
The patent extracts the complex model training process from edge devices and concentrates it on the server. Edge devices only receive pre-trained models and perform simple inference operations, removing the complexity of training algorithms while maintaining real-time processing capability
Solution Approach 2:
The server performs the complex model training in advance and deploys the trained models to edge devices. This preliminary action transfers the computational complexity from edge devices to the server, allowing edge devices to perform only lightweight inference operations in real-time
3Reliability
If inference models are deployed on secure dedicated environments, then security is improved, but scalability is reduced
Solution Approach 1:
The patent segments the inference processing architecture into secure edge devices for local processing and cloud servers for model management. This segmentation allows edge devices to maintain security requirements while enabling scalable model updates and deployments through the cloud infrastructure
4Ease of operation
If server queues are used to manage incoming data, then data organization is improved, but wait times increase when queues become long
Solution Approach 1:
The system performs preliminary inference actions at edge devices before data reaches server queues. This preliminary processing eliminates the need for long server queue waits, as edge devices immediately process time-sensitive data locally while server queues only handle non-urgent data
Data Source
AI summary
Systems and methods for selectively applying inference models on one or more edge devices in response to actual or predicted delays are disclosed. Inference models may be trained and deployed to a server and a first edge device. Sensor data may be received at the server and may also be forwarded to the first edge device. A first inference may be performed on the server by applying the data to the trained inference model to generate a first inference result. The results may be sent to the first edge device. In response to not receiving the first inference result at the first edge device after a delay threshold or in response to a queue length on the server exceeding a threshold, an inference may be performed on the first edge device using the received sensor data. Inference results from the server and the edge device may be combined and reordered.