Generative Model Routing for Entity-Specific Accuracy and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current generative models face challenges with high resource utilization, latency, and accuracy issues due to their large size, while smaller counterparts are less robust and prone to generating inaccurate outputs.
Innovation Solution
A dynamic routing system selects a generative model based on custom selection features specific to the submitting entity, considering both the content of the request and entity-specific preferences, using machine learning models to balance computational efficiency, latency, and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large-sized generative model is used, then accuracy and robustness of generative output are improved, but computational resource usage and latency increase significantly
Solution Approach 1:
The system dynamically selects which generative model to use based on real-time routing decisions. Instead of consistently using the largest model, the system adapts model selection based on request characteristics and entity-specific features, switching between smaller and larger models as needed to balance accuracy and computational efficiency.
Solution Approach 2:
The system changes the parameter configuration by selecting different model sizes (e.g., 7B, 13B, 20B parameters) based on the routing decision. This allows the system to adjust the computational resources allocated to model processing based on the specific needs of each request while maintaining the option for high accuracy when required.
2Reliability
If a large-sized generative model is used, then accuracy and robustness of generative output are improved, but processing latency increases
Solution Approach 1:
The system dynamically adjusts model selection based on real-time routing decisions. By evaluating request characteristics and entity-specific features at runtime, the system can select smaller, faster models when appropriate and reserve larger models for requests that require high accuracy, thereby reducing overall processing latency.
Solution Approach 2:
The routing model performs preliminary analysis of the request and entity features before selecting a generative model. This preliminary routing decision is made quickly based on processed features and selection criteria, enabling the system to prepare and initiate processing with the optimal model beforehand, reducing subsequent latency.
3Productivity
If a smaller generative model is used, then computational efficiency and processing speed are improved, but accuracy and robustness of generative output deteriorate
Solution Approach 1:
The system applies different model quality levels to different requests based on local characteristics. By analyzing entity-specific features and request content, the system determines which requests deserve high-quality large models and which can be handled efficiently with smaller models, optimizing the balance between computational efficiency and accuracy for each specific case.
Solution Approach 2:
The system uses feedback from routing decisions and model performance to continuously improve selection accuracy. By monitoring which models perform best for specific entity types and request patterns, the routing model refines its selection criteria, ensuring that smaller models are chosen when they can provide sufficient accuracy and larger models are reserved when high accuracy is critical.
4Device complexity
If model selection is made without considering entity-specific features, then routing complexity is reduced, but ability to optimize for individual entity needs is lost
Solution Approach 1:
The routing system segments the selection process into distinct stages: extracting entity-specific features, processing request content, evaluating model candidates, and making a final routing decision. This segmentation allows the system to handle entity customization systematically without overwhelming complexity, as each stage processes specific information independently and feeds it to the next stage.
Solution Approach 2:
The routing model acts as an intermediary between the raw request and the generative model selection. It processes entity-specific features and request content, transforms them into routing decisions, and mediates the final model selection. This intermediary layer abstracts the complexity of entity customization from the basic routing mechanism, allowing flexible adaptation without proportionally increasing overall system complexity.
Data Source
AI summary
Implementations disclose selecting, in response to receiving a generative model request and from among multiple candidate generative models, a particular generative model to utilize in generating a response to the generative model request. Various implementations identify an indication of a submitting entity of the generative model request. The particular generative model can be selected based on processing the generative model request and custom selection feature(s) provided by the submitting entity (e.g., provided well in advance of the generative model request). Different submitting entities (e.g., a first and second entities) can have different custom selection features. Accordingly, even if the first and second submitting entities submit the same generative model request, different generative models are selected to process the generative model request, resulting in two different responses, one responsive to the first entity and the other responsive to the second entity.