Multi-Model Inference Cluster with Decoupled API Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning model deployment approaches tightly couple models with API servers, requiring new endpoints for each model change, leading to downtime and inefficiencies in handling large volumes of inference requests.
Innovation Solution
A decoupling mechanism allows any API server in a deployment cluster to run multiple models using a shared runtime environment, with metadata management enabling seamless switching between models without re-instantiating servers, and a generic API endpoint serving inference requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If new API endpoints are created for each model change, then model updates can be deployed, but system downtime occurs and resource utilization decreases
Solution Approach 1:
The patent creates a universal API endpoint that can handle multiple different machine learning models through a decoupling mechanism. Instead of having dedicated endpoints for each model, a single endpoint uses a dispatcher to route requests to appropriate model runners, allowing the same endpoint to serve multiple models dynamically without requiring structural changes or downtime for model updates.
Solution Approach 2:
The patent introduces a dispatcher as an intermediary component between the API endpoint and model runners. This dispatcher receives requests at the generic endpoint, determines which model should handle the request based on input data characteristics, and routes the request to the appropriate model runner. This mediator layer enables flexible model switching without affecting the endpoint itself, eliminating downtime during model updates.
2Reliability
If dedicated API servers are used for each model, then model performance is optimized, but resource utilization efficiency decreases
Solution Approach 1:
The patent makes API servers multi-functional by enabling them to run multiple different machine learning models simultaneously through the decoupling architecture. A single API server can dynamically serve multiple models via the universal endpoint and dispatcher mechanism, eliminating the need for separate dedicated servers for each model and thereby improving resource utilization efficiency while maintaining model performance through isolated model runners.
Solution Approach 2:
The patent merges multiple model serving capabilities into a single API server infrastructure. By combining the universal endpoint, dispatcher, and multiple model runners into one server system, the patent achieves better resource utilization while maintaining the performance isolation needed for different models through the modular architecture of separate model runners that can be loaded and unloaded dynamically.
3Ease of manufacture
If models are tightly coupled with API servers, then deployment is simplified, but scalability and flexibility are reduced
Solution Approach 1:
The patent segments the traditional tight coupling of models and API servers into distinct components: a universal API endpoint, a dispatcher, and separate model runners. This segmentation allows the API server infrastructure to remain simple and stable while the models can be independently loaded, managed, and swapped through the dispatcher mechanism, thereby maintaining deployment simplicity while dramatically improving system flexibility and scalability.
Solution Approach 2:
The patent extracts the model-specific logic and runtime environment from the API server itself, placing models into separate, independently manageable model runners. This extraction allows the API server to maintain its simple, stable structure while models can be freely added, removed, or updated in their own runtime environments without affecting the core API server infrastructure, thus preserving ease of deployment while enhancing flexibility.
4Productivity
If multiple models are deployed on a single cluster, then resource efficiency improves, but model switching complexity increases
Solution Approach 1:
The patent introduces a dispatcher as a mediating layer that simplifies model switching despite multiple models being deployed on a single cluster. The dispatcher handles the complexity of determining which model should process a given request based on input data characteristics, and routes requests to appropriate model runners. This intermediary absorbs the switching complexity, making the model selection and switching process transparent to the API endpoint and clients, thereby enabling efficient multi-model deployment without proportionally increasing operational complexity.
Data Source
AI summary
Systems and methods are provided for servicing inference request by one of multiple machine learning models attached to a deployment cluster. The API server of a deployment cluster is not tightly coupled to any of multiple machine learning models attached to the deployment cluster. Upon receiving an inference request, the deployment cluster can retrieve the configuration parameters, including serialization formatting, for a target model identified in the inference request. The deployment cluster can utilize the retrieved parameters to service the inference request and return the results to a business system application.