Distributed ML Model Hosting with Dynamic Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Loading machine learning models on-the-fly in hosted environments is not feasible due to high latency and memory constraints, making it inefficient to cache all models, especially in systems with a large number of models where only a few receive traffic.
Innovation Solution
Implementing a distributed, multi-tenant hosting system with auto-scaling hosts that cache a subset of models in RAM and use a least frequently used caching strategy for disk storage, allowing more frequently used models to be readily available, and dynamically scaling compute instances based on traffic demand.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If machine learning models are loaded on-the-fly in response to user requests, then memory resources are conserved, but latency increases due to loading operations
Solution Approach 1:
The system pre-loads machine learning models into memory before they are requested by users. Models are loaded in advance and kept ready in memory, so when a request arrives, the model is already available and inference can proceed immediately without loading delays. This preliminary action resolves the contradiction by sacrificing some memory resources to eliminate latency.
Solution Approach 2:
The system dynamically manages model loading and unloading based on request patterns and memory availability. When memory pressure is detected or models are not requested for extended periods, the system unloads models from memory to free resources. Conversely, when models are requested, they are loaded in advance. This dynamic adaptation allows the system to balance memory consumption and latency based on actual usage conditions.
2Speed
If all models are cached in memory, then inference speed is improved, but memory capacity requirements become prohibitive
Solution Approach 1:
Instead of caching all models in memory, the system implements partial caching by loading only a subset of models into memory at any given time. The system prioritizes loading models that are frequently requested or currently needed for inference, while less frequently used models remain on disk. This partial action approach maintains fast inference for active models while avoiding the prohibitive memory capacity requirements of caching everything.
Solution Approach 2:
The system changes the state of model storage from static (all models either in memory or on disk) to dynamic, where models can be moved between disk and memory based on usage patterns. By monitoring request frequencies and memory availability, the system adjusts which models reside in which storage medium, optimizing the balance between inference speed and memory capacity utilization.
3Adaptability or versatility
If a large number of models are hosted, then model variety is increased, but resource utilization becomes inefficient when only a few models receive traffic
Solution Approach 1:
The system dynamically adjusts the set of hosted models based on traffic patterns and demand. When new models are deployed or existing models experience increased traffic, the system automatically loads them into memory. When models experience decreased traffic or are deprecated, the system unloads them to free resources. This dynamic management ensures that resource utilization remains efficient while maintaining the ability to host a large variety of models over time.
Solution Approach 2:
The system implements self-service model management where models effectively manage their own lifecycle in memory. Frequently accessed models automatically maintain their presence in memory through continued usage, while unused models are automatically evicted. This self-service approach allows the system to maintain high model variety without manual intervention, while resources are automatically allocated to only those models that currently serve user needs.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for hosting machine learning models are described. In some instances, a method of receiving a request to perform an inference using a particular machine learning model; determining a group of hosts to route the request to, the group of hosts to host a plurality of machine learning models including the particular machine learning model; determining a path to the determined group of hosts; determining a particular host of the group of hosts to perform an analysis of the request based on the determined path, the particular host having the particular machine learning model in memory; routing the request to the particular host of the group of hosts; performing inference on the request using the particular host; and providing a result of the inference to a requester is performed.