Dynamic Model Server for ML Inference Memory Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing vendors face challenges in providing scalable machine learning (ML) inference services due to the large size of ML models, which require significant memory resources and incur high running costs, especially when serving heterogeneous customer models that are not optimized for web session management technologies.
Innovation Solution
A cloud-based ML inference platform that uses a network API to select and manage model servers dynamically, optimizing memory usage through cache hit/miss ratios and the least-recently-used replacement algorithm, allowing for real-time inference services across multiple, disparate ML models while adjusting the number of servers to balance user experience and costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If ML models are kept loaded in memory to provide good performance, then inference speed and user experience are improved, but memory consumption and running costs increase
Solution Approach 1:
The system dynamically adjusts the number of model servers and their memory allocation based on real-time cache hit/miss ratios and usage patterns. Model servers are provisioned or de-provisioned dynamically to match actual inference workloads, transforming the static memory allocation into a dynamic resource management system that adapts to changing demands.
Solution Approach 2:
The system changes the parameter of memory allocation by adjusting the number of model servers based on cache performance metrics. When cache hit ratios indicate underutilization, the system reduces the number of active model servers, thereby reducing memory consumption while maintaining adequate inference performance through the remaining servers.
2Adaptability or versatility
If multiple model servers are provisioned to handle heterogeneous ML models, then service capability and adaptability are improved, but infrastructure complexity and costs increase
Solution Approach 1:
Model servers are designed as universal, multi-functional units that can load and execute different ML models based on incoming inference requests. Rather than having dedicated servers for each model, a single model server can serve multiple models by dynamically loading them into memory, reducing infrastructure complexity while maintaining the ability to handle heterogeneous ML models.
Solution Approach 2:
The system implements automated resource management where model servers self-regulate their memory usage and the platform automatically provisions or de-provisions servers based on performance metrics. This self-service mechanism reduces operational complexity by eliminating manual intervention in resource allocation while maintaining adaptability to different ML models.
3Loss of energy
If model servers are de-provisioned to reduce costs, then running costs are reduced, but inference latency increases due to model reloading
Solution Approach 1:
The system implements periodic monitoring of cache hit/miss ratios and usage patterns to determine when to provision or de-provision model servers. This periodic assessment allows the system to maintain servers during high-utilization periods (avoiding latency) while reducing server count during low-utilization periods (reducing costs), creating a rhythm that balances performance and cost efficiency.
Solution Approach 2:
The system performs preliminary actions by pre-loading ML models into memory of model servers before they are needed for inference. When a model server is provisioned, models are pre-loaded into memory, so when inference requests arrive, the models are already ready, minimizing latency. This preliminary preparation ensures that cost-saving de-provisioning actions do not result in significant performance penalties.
Data Source
AI summary
Implementations include receiving, by an application programming interface (API) server of a plurality of API servers, a prediction request from a client system, each of the plurality of API servers including a stateless server, selecting, by the API server, a model server from a plurality of model servers based on the prediction request, each of the plurality of model servers including a stateful server, calling, by the API server, the model server to execute inference using a ML model loaded to memory of the model server, receiving, by the API server, an inference result from the ML model, and sending, by the API server, the inference result to the client system.