Multi-Level LLM Routing for Faster, Lower-Load AI Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI generative models, particularly large language models (LLMs), require significant supercomputing efforts to improve response times, accuracies, and reduce computational load, limiting their scalability and expandability.
Innovation Solution
Implement a multi-level generative AI system utilizing a network of smaller, specialized LLMs (h-LLMs) with varying levels of accuracy, precision, and training sets, combined through techniques like bagging, boosting, and distributed architectures to optimize computational efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single large LLM with many parameters is used to handle all tasks, then the model can perform diverse language processing tasks, but the computational load and cost increase significantly
Solution Approach 1:
The patent divides a single large LLM into multiple smaller specialized LLMs, each trained to handle specific task categories (e.g., summarization, translation, question answering). This segmentation reduces the computational load for each individual task while maintaining overall system versatility through the collection of specialized models.
Solution Approach 2:
The patent introduces a task router or intermediary component that receives incoming requests, classifies the task type, and routes the request to the appropriate specialized LLM. This intermediary enables efficient task distribution without requiring a single large model to handle all tasks, thereby reducing computational overhead.
2Adaptability or versatility
If a single large LLM is used for all tasks, then comprehensive task handling is achieved, but response time increases due to processing overhead
Solution Approach 1:
By segmenting the large LLM into smaller specialized models, each model can process its designated tasks more quickly without the overhead of a larger model architecture. This segmentation directly improves response time for specific task types while maintaining comprehensive task handling capability across the system.
Solution Approach 2:
The patent implements pre-processing steps including task classification and model selection before actual LLM processing begins. This preliminary action ensures that the most appropriate specialized model is selected in advance, avoiding unnecessary processing delays and improving overall response time.
3Device complexity
If traditional single-model LLM architecture is used, then implementation is simpler, but scalability and cost-effectiveness are reduced
Solution Approach 1:
The patent segments the LLM system into modular specialized models that can be independently deployed, scaled, and managed. This modular architecture improves scalability by allowing selective deployment of only the needed models for specific tasks, reducing overall system cost while maintaining flexibility and productivity.
Data Source
AI summary
A method for improving responses to large language model (LLM) prompts including receiving at an input broker a request input from a user including an LLM prompt, deriving a search query from the LLM prompt, searching a plurality of documents using the search query to identify a first subset of documents and a second subset of documents, generating a first answer using a first context-specific LLM, where the first context-specific LLM uses the first subset of documents as the context and the LLM prompt as the prompt, generating a second answer using a second context-specific LLM, where the second context-specific LLM uses the second subset of documents as the context and the LLM prompt as the prompt, providing each of the first answer and the second answer to an output broker, determining a primary result at the output broker, and transmitting the primary result to the user.