Conversational Response Retrieval to Cut Direct LLM Processing Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language model (LLM)-based chatbots incur high processing and storage costs, making them prohibitive in certain situations, and existing systems are limited by their reliance on training data, which can restrict their applicability.
Innovation Solution
A system that analyzes conversational user inputs to identify semantically similar prompts from previous interactions with LLM-based chatbots, generating responses that emulate the LLM-based chatbot's output without directly sending new prompts to the LLM, thereby reducing computational and monetary costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LLM-based chatbots are used to provide information to users, then the quality and versatility of responses is improved, but processing and storage costs increase significantly
Solution Approach 1:
The system creates a copy mechanism by storing historical interactions between users and the LLM-based chatbot, then using these stored copies to respond to new queries without re-engaging the LLM. This allows the system to replicate LLM response quality for common queries while avoiding the high processing costs of direct LLM engagement.
Solution Approach 2:
The system performs preliminary actions by pre-storing historical interactions and responses in a database before they are needed. When a user submits a query, the system first checks the stored historical data for matches, only engaging the LLM when necessary. This preliminary preparation reduces the frequency of expensive LLM processing operations.
2Adaptability or versatility
If LLM-based chatbots are deployed for customer service, then customer support capability is enhanced, but monetary costs become prohibitive
Solution Approach 1:
The system implements self-service by enabling the chatbot to automatically respond to user queries using stored historical interactions without requiring continuous LLM engagement. The database system autonomously matches incoming queries with historical data and generates responses, eliminating the need for expensive real-time LLM processing for routine customer service tasks.
3Loss of energy
If traditional chatbots with predetermined responses are used, then processing costs are reduced, but adaptability to new situations is limited
Solution Approach 1:
The system achieves universality by combining the low-cost structure of traditional chatbots with the adaptability of LLM-based systems. By storing diverse historical interactions in a database and using pattern matching, the system can handle multiple types of queries and situations with a single unified architecture, rather than requiring separate specialized systems for different functions.
Data Source
AI summary
Database systems and methods are provided for managing usage of large language models (LLMs). One method involves determining a numerical representation of a conversational input to a user interface, identifying a semantically similar subset of prior conversational inputs based at least in part on the numerical representation of the conversational input, and determining numerical representations of respective conversational responses generated by a language model responsive to the respective prior conversational input of the semantically similar subset. When the numerical representations of the respective conversational responses satisfy a semantic similarity threshold, the method automatically generates an automated response to the conversational input based at least in part on one or more prior conversational responses and automatically provides the automated response to the user interface responsive to the conversational input.