AI Agent Training With Preference Learning for Fewer Hallucinations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current large language model (LLM)-based AI agents perform poorly across different tasks and generate incorrect results, even with retrieval-augmented generation (RAG), due to challenges in handling diverse tasks and generating hallucinations.
Innovation Solution
A training framework is developed that includes the generation of synthetic training data with additional roles such as 'thought' and 'observation' to enhance multi-turn reasoning, and employs preference learning to improve response accuracy by using positive and negative samples, along with supervised fine-tuning on various tasks like non-coherent distractors, coherent distractors, abstinence of unanswerables, extraction and citation, diversification, summarization, structured data, and function calling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If retrieval-augmented generation (RAG) is used to improve response accuracy, then information retrieval capability is enhanced, but hallucination generation increases
Solution Approach 1:
The patent introduces an intermediary verification mechanism where the language model's generated responses are cross-checked against the retrieved context information. This intermediary step identifies and flags potential hallucinations by comparing generated content with source material, thereby reducing hallucination generation while preserving RAG's information retrieval capability
Solution Approach 2:
The system implements feedback loops where the model's performance on hallucination detection is continuously monitored and used to refine the verification process. The feedback mechanism adjusts the balance between utilizing retrieved information and maintaining factual accuracy, reducing hallucinations while preserving response accuracy
2Ease of manufacture
If standard training data is used to train the language model, then training simplicity is maintained, but performance across diverse tasks deteriorates
Solution Approach 1:
The patent creates a universal training framework that generates synthetic training data capable of covering multiple task types and domains. The synthetic data generation process incorporates diverse task scenarios, reasoning patterns, and contextual variations, enabling the model to achieve broad task adaptability while maintaining a unified training pipeline
Solution Approach 2:
The system performs preliminary generation of synthetic training data that pre-encodes diverse task scenarios and reasoning patterns. By preparing this comprehensive training corpus in advance, the model is pre-equipped to handle various tasks without requiring complex post-training adjustments, thus maintaining training simplicity while improving versatility
3Reliability
If complex training frameworks are implemented to improve reasoning capability, then multi-hop reasoning performance is enhanced, but training complexity increases
Solution Approach 1:
The patent segments the complex reasoning training process into distinct synthetic data generation stages, each focusing on specific reasoning patterns (e.g., single-hop, multi-hop, causal reasoning). This segmentation allows the model to learn different reasoning skills through targeted synthetic examples while keeping the overall training framework manageable and systematic
Data Source
AI summary
Embodiments described herein provide a method for training a neural network based language model (LM). The method includes receiving, via a data interface, a training dataset including pairs of user queries and ground-truth responses; generating, via the LM, a plurality of responses based on a query from the training dataset; identifying, from the plurality of responses, a first response having a first similarity metric value below a threshold, based on a similarity metric associated with a corresponding ground-truth response from the training dataset; training the LM using the first response as a negative sample and a second response as a positive sample such that the LM after training is more likely to generate the positive sample and less likely to generate the negative sample; receiving, via a user interface, a query; and generating a response to the query via the trained LM.