Container Selection for Machine Learning Model Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine-learning serving infrastructures face challenges in scaling to accommodate multiple machine-learning models due to resource limitations and the need for rebalancing resources, as well as the inefficiency of updating software code when adding new clusters of serving containers.
Innovation Solution
A system that uses container information to select the appropriate serving containers for executing models by identifying available containers through consistent hashing and round-robin strategies, allowing for efficient loading and execution of machine-learning models across clusters, thereby scaling to host more models than individual containers can handle and enabling on-demand resource allocation without code updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a one-to-one relationship is used where each serving container hosts only one machine-learning model, then model execution reliability is improved, but resource utilization efficiency deteriorates
Solution Approach 1:
A single serving container is designed to host and execute multiple machine-learning models simultaneously, transforming the container from a single-function to multi-function unit. The container includes a model selector that can choose which model to execute based on incoming requests, enabling one container to serve multiple models and thereby improving resource utilization while maintaining execution reliability through isolated execution environments.
2Productivity
If multiple machine-learning models are hosted in a single serving container, then resource utilization efficiency is improved, but container selection complexity increases
Solution Approach 1:
A routing container is introduced as an intermediary component that manages the complexity of selecting which serving container should execute a given machine-learning model. The routing container receives requests, determines the appropriate model, and routes the request to the correct serving container. This mediator approach simplifies the selection process by centralizing the decision-making logic and avoiding the need for complex client-side selection algorithms.
Solution Approach 2:
The system introduces a new dimension of organization by using consistent hashing based on model identifiers to map models to specific serving containers. Instead of managing container selection through complex hierarchical or administrative dimensions, the system uses a mathematical hashing dimension that automatically distributes models across containers in a deterministic and balanced manner, reducing selection complexity.
3Stability of the object's composition
If consistent hashing is used to select serving containers, then load distribution uniformity is improved, but selection algorithm complexity increases
Solution Approach 1:
The system pre-computes and stores hash values for each machine-learning model identifier, creating a mapping table that directly associates model IDs with their corresponding serving container assignments. This copying approach transforms the complex consistent hashing algorithm into a simple table lookup operation, maintaining uniform load distribution while dramatically reducing the computational complexity of the selection process during runtime.
Data Source
AI summary
Using container information to select containers for executing models is described. A system receives a request from an application and identifies a version of a machine-learning model associated with the request. The system identifies a set of each serving container corresponding to the machine-learning model from a cluster of available serving containers associated with the version of the machine-learning model. The system selects a serving container from the set of each serving container corresponding to the machine-learning model. If the machine-learning model is not loaded in the serving container, the system loads the machine-learning model in the serving container. If the machine-learning model is loaded in the serving container, the system executes, in the serving container, the machine-learning model on behalf of the request. The system responds to the request based on executing the machine-learning model on behalf of the request.