LLM Fine-Tuning with Search Feedback for Factual Grounding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) often generate factually incorrect responses (hallucinations) due to the lack of grounding in factual information, and techniques to address this, such as retrieving search results, incur significant computational costs and resource overhead.
Innovation Solution
Fine-tuning LLMs using reinforcement learning with a reward model trained on search engine results to generate more factually accurate responses, leveraging reinforcement learning techniques like policy gradient, Q-learning, and actor-critic methods to optimize responses based on factual correctness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If search results are retrieved and provided as part of the prompt to the LLM, then factual accuracy of responses is improved, but computational cost and resource overhead increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training the LLM on search engine results and factual information during the fine-tuning phase. This allows the model to internalize factual knowledge beforehand, eliminating the need to retrieve and process search results at runtime for every query, thus reducing computational cost while maintaining factual accuracy
Solution Approach 2:
The patent uses copying by creating a refined version of the LLM through fine-tuning with search engine conditioned data. Instead of repeatedly accessing external search results, the model copies factual knowledge into its parameters, allowing it to generate factually accurate responses without ongoing dependency on external search infrastructure
2Reliability
If search engine results are generated at runtime, then responses are more factually grounded, but processing time increases due to lengthy prompt processing
Solution Approach 1:
The patent applies preliminary action by pre-training the LLM on search engine results and factual information during the fine-tuning phase. This allows the model to internalize factual knowledge beforehand, eliminating the need to retrieve and process search results at runtime for every query, thus reducing computational cost while maintaining factual accuracy
Solution Approach 2:
The patent enables continuity of useful action by making the LLM self-sufficient in generating factually grounded responses without intermittent external search queries. The model continuously operates using its internalized knowledge, eliminating the stop-start nature of external search retrieval and maintaining steady response generation throughput
3Adaptability or versatility
If the LLM is trained on enormous amounts of diverse data, then its ability to perform various NLP tasks is improved, but it generates more hallucinations due to lack of grounding in factual information
Solution Approach 1:
The patent applies local quality by specifically enhancing the factual grounding aspect of the LLM through targeted fine-tuning with search engine conditioned data. Instead of retraining the entire model, the approach locally improves the factual accuracy property while preserving the model's existing NLP capabilities learned from diverse training data
Solution Approach 2:
The patent uses parameter changes by adjusting the LLM's parameters through fine-tuning with a specialized dataset comprising search engine results and factually grounded responses. This modifies the model's parameter space to favor factually accurate outputs while maintaining its versatility across different NLP tasks
Data Source
AI summary
Various implementations are directed towards fine-tuning a large language model (LLM) using search engine feedback (e.g., responsive content generated based on a reference source material such as a set of search engine results). Additionally or alternatively, a supervision signal can be generated based on comparing search engine conditioned LLM output with unconditioned LLM output. In many implementations, the supervision signal(s) can be used in training a reward model using reinforcement learning, where the trained reward model can be used in fine-tuning the LLM.