Semantic FAQ Cache for Reducing Repeated LLM Query Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The widespread use of large language models (LLMs) like ChatGPT and GPT-4 is costly in terms of computing resources due to the overhead of repeatedly answering similar user queries, making traditional FAQ pages obsolete as LLMs can provide more natural responses, but continuous querying is resource-intensive.
Innovation Solution
Implement a caching mechanism that converts user queries into embeddings, performs semantic searching against a cache of question and answer pairs, and returns stored answers when similar, while using the LLM only when no cache hit is found, caching new query-answer pairs for future reference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs are used to answer user queries directly, then response quality and natural language understanding are improved, but computing resource consumption increases
Solution Approach 1:
The system performs preliminary action by converting user queries into embeddings and performing semantic searching against a cache of question-answer pairs before invoking the LLM. This pre-check mechanism identifies frequently asked questions and their answers in advance, allowing the system to return cached results without consuming LLM computing resources, thus resolving the contradiction between response quality and resource consumption
Solution Approach 2:
The patent introduces an intermediary component - the semantic search cache system - that sits between the user query and the LLM. This intermediary converts queries to embeddings, searches for matching cached answers, and only passes new or uncached queries to the LLM. This mediator reduces direct LLM usage while maintaining response quality for cached questions, effectively resolving the resource consumption vs. response quality contradiction
2Ease of operation
If LLMs handle all user queries, then user interaction quality is improved, but system overload occurs
Solution Approach 1:
The system segments user queries into two categories: cached questions with pre-computed embeddings and answers, and new or uncached queries requiring LLM processing. By dividing the query handling workload in this manner, the system maintains high-quality user interactions for cached questions through fast semantic search while preventing system overload by limiting LLM invocations to only necessary cases
Solution Approach 2:
The patent changes the operational parameters of the query handling system by introducing embedding-based semantic similarity thresholds. Instead of uniformly processing all queries through the LLM, the system uses parameter-based filtering (similarity scores, cache hit/miss conditions) to dynamically route queries, thereby maintaining interaction quality while reducing system overload
Data Source
AI summary
In one embodiment, a method for a cache-generated frequently asked questions page includes converting a received query into a set of embeddings and performing a semantic searching operation based on information contained in the set of embeddings against a cache of question and answer pairs. The method further includes returning a stored answer to the received query responsive to a determination that a particular question and answer pair in the cache of question and answer pairs meets a threshold similarity level for the information contained in the set of embeddings, the stored answer derived from the particular question and answer pair and performing a large language model operation to generate an answer to the query responsive to a determination that no question and answer pair in the cache of question and answer pairs meets the threshold similarity level for the information contained in the set of embeddings.