LLM Query Response Verification via Parallel Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users lack a mechanism to verify the accuracy of query responses generated by large language models (LLMs), as the responses act as a 'black box' without providing sources for the information.
Innovation Solution
A technique is implemented where a user receives subtopics and keywords from an LLM, which are then used to perform parallel searches across multiple search engines. The search results are evaluated based on keyword frequency, and the top-ranked sentences are selected to generate a query response that includes links to the original sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If LLM generates query responses directly, then response generation speed is improved, but accuracy verification capability deteriorates
Solution Approach 1:
The system segments the LLM response into multiple subtopics and generates separate search queries for each subtopic. This segmentation allows parallel verification of different parts of the response, maintaining speed while improving verification capability through distributed validation across multiple search engines.
Solution Approach 2:
The system performs preliminary actions by generating search queries from LLM subtopics and keywords before final response assembly. Search engines execute these queries in parallel to pre-verify information accuracy, and results are evaluated against keywords before being incorporated into the final response, ensuring verification happens ahead of time.
2Device complexity
If LLM acts as black box, then system complexity is reduced, but information transparency deteriorates
Solution Approach 1:
Search engines act as intermediaries between the LLM black box and the user. They provide the transparency layer by verifying LLM-generated information through independent searches and providing source links, allowing users to trace information origins without exposing the complex LLM internal workings.
Solution Approach 2:
The system extracts source verification information from the LLM response process by generating separate search queries and retrieving original source links. This extracted verification data is presented to users alongside LLM responses, separating the transparency function from the LLM's black box operation.
Data Source
AI summary
A query is received from a client device. A large language model is prompted to generate a plurality of subtopics on the query and to generate a corresponding plurality of keywords for each of the plurality of subtopics. One or more search engines are utilized to perform a plurality of searches utilizing the plurality of subtopics and the corresponding plurality of keywords received from the large language model. A plurality of responses corresponding to the plurality of subtopics and the corresponding plurality of keywords is received from the one or more search engines. The plurality of responses is evaluated based on the corresponding plurality of keywords.