Edge Model Pooling for Concurrent Inference Under Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Operating machine learning models on edge devices is challenging due to limited compute, memory, and connectivity, and updating models requires disruptive changes to the application, making it difficult to monitor and update models without disrupting operations.
Innovation Solution
An edge manager is deployed on edge devices to optimize, run, and update machine learning models across fleets of devices without changing the application, allowing for model updates to be transparent to the application and decoupled from any application updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are run on edge devices with limited resources, then predictions can be made in real-time and user privacy is preserved, but the compute capacity and memory available for running multiple complex models is insufficient
Solution Approach 1:
Multiple model instances are merged into a single shared memory space with a unified allocator. Instead of each model having dedicated memory, the system combines memory resources and uses a single allocator to manage all model instances, reducing total memory requirements and enabling more models to run concurrently on resource-constrained edge devices.
Solution Approach 2:
A single allocator serves multiple model instances universally, rather than having dedicated allocators for each model. This universal allocator can service any model instance, reducing the overhead of multiple allocation mechanisms and improving memory utilization efficiency across all models running on the edge device.
2Ease of operation
If models are hard-coded into the application, then inference requests can be made directly, but updating models requires rebuilding and updating the entire application which disrupts operations
Solution Approach 1:
The system segments the model management functionality from the application code by introducing a model pool and edge manager. Models are stored as separate, interchangeable instances in the pool, allowing the application to request inference without being coupled to specific model implementations. This enables model updates independently of application deployment, maintaining operational continuity.
Solution Approach 2:
An edge manager acts as an intermediary between the application and the model pool. Instead of the application directly containing model code, it communicates through the edge manager which manages the model pool. This intermediary layer decouples the application from specific model versions, allowing models to be updated in the pool without requiring application rebuilds or disruptions.
3Productivity
If multiple model instances are loaded into memory, then concurrent predictions can be performed, but the memory footprint increases significantly
Solution Approach 1:
Multiple model instances are merged into a shared memory space where they can coexist and be serviced concurrently. The unified allocator efficiently manages memory allocation across all instances, reducing the total memory footprint compared to having completely separate memory spaces for each model while still enabling concurrent predictions.
Solution Approach 2:
The system changes the memory allocation parameters from dedicated, static allocation per model to dynamic, shared allocation. This allows memory to be allocated and deallocated based on actual usage patterns and concurrency requirements, optimizing the memory footprint while maintaining the ability to perform concurrent predictions when needed.
Data Source
AI summary
In some examples, a method includes receiving a request to load a first machine learning model on an edge device, wherein the request includes an external identifier of the first machine learning model. The method further includes loading at least one instance of the first machine learning model onto the edge device, wherein the first machine learning model is loaded into a model pool having at least a second machine learning model using the same external identifier as the first machine learning model.