Intent-Based LLM Arbitration for Faster, Lower-Cost Prompt Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face issues such as slow response times, outdated information, hallucinations, and high costs due to token-based pricing, particularly when dealing with user prompts that require specific subject matter knowledge.
Innovation Solution
A system and method that dynamically processes user prompts using a prompt processing server to select appropriate information resource engines based on determined intents, leveraging LLMs for general knowledge and specialized engines for specific information, and employing parallel and nested processing of token groups to enhance response quality and reduce costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If LLMs are used to process all user prompts, then general knowledge and conversational interactions are provided, but response time is slow and cost is high
Solution Approach 1:
The system segments prompt processing into two paths: intent classification (handled by LLM) and information retrieval (handled by specialized engines). This segmentation allows the LLM to quickly identify prompt types and route them appropriately, improving overall response speed while maintaining accurate handling of complex queries.
Solution Approach 2:
An intermediary intent classification layer is introduced between the user prompt and the information retrieval system. This intermediary component analyzes the prompt intent and dynamically selects the appropriate information resource engine, acting as a mediator that optimizes the processing pipeline and reduces unnecessary LLM computations.
2Reliability
If LLMs are used to process all user prompts, then comprehensive information is provided, but cost increases due to token consumption
Solution Approach 1:
The system applies partial action by using the LLM only for intent classification and complex reasoning, while routing straightforward information queries to specialized engines. This partial deployment of LLM capabilities reduces token consumption and computational cost while maintaining high reliability for prompts that require LLM expertise.
Solution Approach 2:
The system changes the parameter of prompt processing by dynamically adjusting which engine handles each prompt based on intent classification. This parameter change allows the system to optimize cost-performance ratio by selecting the most efficient processing path for each specific prompt type.
3Adaptability or versatility
If LLMs are used to provide specific subject matter information, then general responses are given, but accuracy and up-to-date information are compromised
Solution Approach 1:
The system applies local quality by matching specific prompt intents to specialized information resource engines that have deep expertise in particular domains. Instead of using a single general-purpose LLM for all queries, the system locally optimizes information retrieval by selecting domain-specific engines for each prompt type, improving accuracy while maintaining versatility.
Solution Approach 2:
The intent classification layer provides universality by handling all prompt types through a single LLM component, while the information retrieval layer provides multi-functionality through multiple specialized engines. This architecture allows the system to maintain broad knowledge coverage while delivering high accuracy for specific domains through specialized processing.
Data Source
AI summary
Systems and methods are provided for processing prompts to a large language model based on a corresponding intent of a received prompt. The systems and methods select, based on determined corresponding intents, from a plurality of information resource engines to process the received prompts.