LLM Answer Verification via External Search and Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models struggle with incorporating and operating large language models due to their size and complexity, leading to difficulties in evaluating the credibility and appropriateness of generated answers.
Innovation Solution
An information processing system comprising three components: a first component for receiving and transferring question documents, a second component for processing using a large language model to generate draft answers, and a third component for examining draft answers using a search engine and database to evaluate their credibility and appropriateness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large language model is used to generate answers, then the capacity and natural language processing capability are significantly increased, but it becomes difficult to incorporate and operate the model at one's own facilities and expense
Solution Approach 1:
The system divides the language model into multiple smaller components (first language model and second language model) that can be operated separately. This segmentation allows each component to be more manageable in terms of computational resources while collectively providing comprehensive natural language processing capabilities.
Solution Approach 2:
The patent introduces an intermediary evaluation mechanism that assesses the reliability of generated answers without requiring the entire large language model to be deployed. This intermediary layer enables verification of model outputs while reducing the operational burden of running the full model.
2Ease of operation
If external language models are used via service, then the operational complexity is reduced, but the control and customization capability over the model is limited
Solution Approach 1:
By segmenting the language model into multiple smaller models that can be independently deployed and controlled, the system achieves both ease of operation (each small model is simpler to manage) and adaptability (each model can be customized for specific tasks or domains).
Solution Approach 2:
The system creates a universal framework that can accommodate multiple language models with different specializations. This multi-functional approach allows the system to leverage various models for different purposes while maintaining a unified evaluation and deployment platform.
3Productivity
If the language model generates answers based on its trained data, then the generation speed is fast, but the accuracy of information not included in the training data cannot be ensured
Solution Approach 1:
The system implements a feedback mechanism where generated answers are evaluated by an evaluation model that checks for hallucinations and verifies information accuracy. This feedback loop maintains fast generation speeds while ensuring reliability by identifying and correcting inaccurate information.
Solution Approach 2:
The evaluation model performs preliminary verification of generated answers before they are finalized. By conducting this preliminary check, the system ensures accuracy for information outside the training data while maintaining efficient generation throughput through parallel processing.
Data Source
AI summary
A novel information processing system that is highly convenient, useful, or reliable is provided. The information processing system is composed of three components. A first component has a function of receiving a question document and providing an answer document. A second component receives a prompt, creates a draft answer with the use of a large language model, and transfers the draft answer to a third component. The third component creates a question document and obtains a search result from a database. The third component examines the draft answer on the basis of the search result and transfers the draft answer to the first component when the draft answer is true. The third component creates the answer document with the use of the search result and transfers the answer document when the draft answer is false. This system is intended to provide an appropriate answer to a question.