Edge Manager Model Update via Shared Handle
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Operating machine learning models on edge devices is challenging due to limited compute, memory, and connectivity, making it difficult to monitor and update models without disrupting applications, especially when model drift occurs and updates require changes to the application or device firmware.
Innovation Solution
An edge manager is deployed on edge devices to optimize, run, and update machine learning models across fleets of devices, allowing for inference handling, model monitoring, and updates without changing the application, by managing model lifecycles, batching requests, and using shared memory and buffer copies to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are deployed on edge devices with limited resources, then real-time predictions and data privacy are improved, but model monitoring and updating become difficult without disrupting applications
Solution Approach 1:
The system segments the model lifecycle management into distinct components: model training, model packaging, model deployment, and model monitoring. This allows independent updates of models without requiring application changes, resolving the contradiction between maintaining real-time prediction reliability and enabling easy model updates on resource-constrained edge devices
Solution Approach 2:
An intermediary model management system is introduced between the application and the ML models. This intermediary handles model updates, monitoring, and deployment automatically, allowing model changes without application disruption while maintaining real-time prediction capabilities on edge devices
2Reliability
If multiple complex neural network models are run on edge devices, then model accuracy and functionality are improved, but resource consumption increases
Solution Approach 1:
The system dynamically manages model deployment by loading only necessary models based on current inference requirements and unloading unused models. This dynamic allocation allows multiple complex models to be available without continuously consuming resources, balancing model accuracy with energy consumption on edge devices
Solution Approach 2:
The system changes operational parameters by adjusting model precision requirements based on available resources. When resources are constrained, models can switch between different precision modes (e.g., full precision, quantized), allowing multiple models to run while managing energy consumption and maintaining adequate accuracy
3Reliability
If model updates are implemented on edge devices, then model drift is addressed and accuracy is improved, but application disruption occurs
Solution Approach 1:
Models are pre-trained and packaged in advance with all necessary dependencies and configurations. This preliminary preparation allows models to be deployed and updated without requiring application changes or disruptions, as the pre-packaged models contain everything needed for immediate execution on edge devices
Solution Approach 2:
The system creates and manages copies of models in different states (training, staging, production). Updates are performed by creating new model copies and swapping them into the production environment without disrupting the running application. This copying mechanism enables continuous model improvement while maintaining application stability
Data Source
AI summary
Techniques for machine learning prediction using edge devices are described. In some examples a method of use includes receiving a request to load a second model onto an edge device while a first model has already been loaded on the edge device, wherein the second model and the first model share an external handle; loading at least one instance the second model into memory of the edge device; and after the second model has been loaded into memory of the edge device, directing a prediction request to the shared external handle to the second model instead of the first model.