LLM Query Routing by Category for Response Quality and Compute Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies in selecting the most suitable large language models (LLMs) for user queries due to variations in architecture and training data, leading to suboptimal performance and redundant computational resource usage.
Innovation Solution
An online system categorizes user queries and maps them to preferred LLMs based on performance data, using a dataset and analytics engine to dynamically select the best-performing model for each query category, thereby improving response quality and reducing computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single LLM is used for all queries, then device complexity is reduced, but response quality deteriorates due to model- query mismatch
Solution Approach 1:
The patent segments queries into different categories (e.g., coding, creative writing, data analysis) and assigns specific LLMs to each category based on their specialized strengths. This segmentation allows each query to be handled by the most suitable model, improving response quality without requiring a single monolithic model to handle all tasks.
Solution Approach 2:
The system dynamically selects which LLM to invoke based on real-time query analysis. The query classification module continuously monitors incoming queries and routes them to appropriate models, making the system adaptable to different task requirements without static model assignments.
2Reliability
If multiple LLMs are deployed, then response quality improves through specialized models, but computational resource usage increases
Solution Approach 1:
Instead of maintaining a large pool of models ready for all tasks, the system loads only the necessary models when needed based on query category. This partial action approach ensures high response quality for specific tasks while minimizing computational resource consumption by avoiding idle model instances.
Solution Approach 2:
The query classification module acts as an intermediary between the user query and the LLM selection process. It analyzes queries, determines the appropriate model category, and routes to the correct model, thereby optimizing resource allocation and reducing unnecessary computational overhead from deploying all models simultaneously.
3Reliability
If queries are routed to appropriate LLMs based on categories, then response quality improves, but device complexity increases due to routing mechanisms
Solution Approach 1:
The query classification module serves multiple functions: it categorizes queries, determines model suitability, and routes to appropriate models. This multi-functionality consolidates routing logic into a single component, reducing overall system complexity while maintaining high response quality through intelligent query-model matching.
Data Source
AI summary
An online system receives a user query for execution by at least one of a set of generative artificial intelligence (AI) models. The online system assigns the user query to one or more query categories of a set of query categories. The online system accesses a dataset stored in a database. For each query category, the dataset stores a preferred generative AI model for the query category among the set of generative AI models. The online system selects a preferred generative AI model for the user query from the database based on the one or more query categories assigned to the user query. The online system provides a prompt to a model serving system hosting the selected generative AI model. The online system receives, from the model serving system, a response to the user query generated by executing the prompt.