Multi-Tenant Chatbot Serve Pools with Pre-Created Deployments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional chatbot systems face significant delays in creating new containers for serving user queries, leading to query failures due to timeouts, which negatively impact user experience.
Innovation Solution
A multi-tenant serve pool system is implemented, where a pool of deployments with both serving and model-manager containers is maintained, allowing for rapid selection and loading of machine-learning models to serve queries, reducing query time to a few seconds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a new container is created to host the machine-learning model for a chatbot query, then the system can serve the query, but the query is likely to fail due to timeout because container creation takes tens of seconds
Solution Approach 1:
The system pre-creates and maintains a pool of serving containers that are ready to host machine-learning models before queries arrive. When a query is received, the system can immediately assign an existing container from the pool to the query, eliminating the need to create a new container at query time and avoiding timeout failures.
Solution Approach 2:
The system merges multiple chatbot models into shared serving containers through a multi-tenant architecture. A single container can host multiple machine-learning models, allowing the system to serve multiple chatbots with a limited pool of containers, thereby reducing the frequency of container creation while maintaining service availability.
2Reliability
If multiple chatbots are served with dedicated containers, then each chatbot has isolated resources, but the system requires many containers which increases complexity and resource consumption
Solution Approach 1:
The system implements multi-tenant serving containers that can host multiple machine-learning models simultaneously. Each serving container is designed to be universal and can accommodate different chatbot models, allowing a single container to serve multiple chatbots and reducing the total number of containers needed in the system.
Solution Approach 2:
The system segments chatbot models into separate model artifacts that can be independently loaded and unloaded from serving containers. This segmentation allows flexible allocation of models to containers based on demand, enabling the system to maintain service availability with fewer physical containers by dynamically assigning model segments to available container resources.
Data Source
AI summary
Techniques are disclosed for providing a scalable multi-tenant serve pool for chatbot systems. A query serving system (QSS) receives a request to serve a query for a new skillbot. The QSS comprises a plurality of deployments, each of which is configured to host a plurality of machine-learning models, each machine-learning model being associated with a skillbot, each deployment including a serving container and a model manager container that hosts a model manager, the serving container including a plurality of sub-containers, each of which hosts one of the machine-learning models downloaded by the model manager. The QSS selects a first deployment to be assigned to the new skillbot based on a first criterion, and loads the machine-learning model associated with the new skillbot into the first deployment. The machine-learning model is trained to serve the query for the new skillbot. The query is served using the machine-learning model.