Conversational Response Retrieval to Cut LLM Processing Costs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language model (LLM)-based chatbots incur high processing and storage costs, making them prohibitive in certain situations, despite their ability to provide wide-ranging information.
Innovation Solution
A system that analyzes semantically similar conversational prompts to generate responses mimicking LLM-based chatbots, bypassing direct use of LLMs by encoding user inputs, comparing them to prior responses, and generating responses based on similarity thresholds, thus reducing computational and monetary costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LLM-based chatbots are used to provide wide-ranging information, then response quality and information coverage are improved, but processing and storage costs increase significantly
Solution Approach 1:
The patent creates simplified copy models that replicate the essential response-generation capabilities of LLMs without requiring full LLM infrastructure. These copy models are trained on LLM-generated responses and can produce similar outputs for common queries, significantly reducing processing costs while maintaining acceptable response quality and information coverage.
2Adaptability or versatility
If LLM-based chatbots are used to provide wide-ranging information, then response quality and information coverage are improved, but storage requirements increase
Solution Approach 1:
Instead of storing and processing the full LLM model weights and parameters which require substantial storage, the patent stores simplified copy models and training datasets. These copies occupy significantly less storage space while preserving the essential functionality needed to generate informative responses across diverse topics.
3Use of energy by moving object
If simplified response generation methods are used to reduce costs, then processing and storage costs decrease, but response quality and adaptability may deteriorate
Solution Approach 1:
The patent performs preliminary actions by pre-training copy models on comprehensive datasets generated by LLMs. This pre-training equips the simplified models with broad knowledge and adaptability before deployment, enabling them to handle diverse queries effectively without requiring expensive real-time LLM processing, thus maintaining response quality while reducing operational costs.
Data Source
AI summary
Database systems and methods are provided for managing usage of large language models (LLMs). One method involves determining a numerical representation of a conversational input to a user interface, identifying a semantically similar subset of prior conversational inputs based at least in part on the numerical representation of the conversational input, and determining numerical representations of respective conversational responses generated by a language model responsive to the respective prior conversational input of the semantically similar subset. When the numerical representations of the respective conversational responses satisfy a semantic similarity threshold, the method automatically generates an automated response to the conversational input based at least in part on one or more prior conversational responses and automatically provides the automated response to the user interface responsive to the conversational input.