Abstractive Summarization with Contrastive Parameter Ensembling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural abstractive summarization systems frequently hallucinate information due to errors in training data, and existing methods to address this issue are inefficient and costly.
Innovation Solution
A document summarization framework called Contrastive Parameter Ensembling (CaPE) that combines parameters from a base model, an expert model, and an anti-expert model, trained on clean and noisy data subsets, to reduce factual errors in summaries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural abstractive summarization models are trained on available training data, then they can generate summaries efficiently, but they hallucinate (generate false information) at a high frequency
Solution Approach 1:
The training dataset is segmented into clean samples and noisy samples based on factual metrics. The model is trained separately on clean samples to learn accurate summarization patterns, and then fine-tuned on noisy samples with modified objectives to learn to avoid hallucinations. This segmentation allows the model to maintain efficiency while improving factual accuracy.
Solution Approach 2:
The training approach combines two different training strategies: standard summarization training on clean data and hallucination-aware training on noisy data. This composite training methodology creates a model that benefits from both accurate language generation and hallucination resistance, resolving the contradiction between efficiency and reliability.
2Reliability
If training data is cleaned to remove factual errors, then hallucination frequency decreases, but the process is time-inefficient and costly
Solution Approach 1:
The system automatically identifies and categorizes training samples as clean or noisy using factual metrics without requiring manual inspection or cleaning. The model itself learns to distinguish between accurate and inaccurate information patterns during training, eliminating the need for time-consuming human data cleaning processes while maintaining high factual accuracy.
3Productivity
If all training samples are used including noisy ones, then training speed increases, but hallucination frequency increases
Solution Approach 1:
The training process is divided into two phases: first training on clean samples to establish baseline summarization capabilities, then fine-tuning on noisy samples with specialized objectives. This segmented approach allows the model to learn from all available data quickly while maintaining factual accuracy through the structured training sequence.
Solution Approach 2:
The model is preliminarily trained on clean samples before being exposed to noisy samples. This preliminary action establishes a solid foundation of accurate summarization patterns, which then serves as a basis for learning to handle noisy data without compromising factual accuracy, thereby maintaining training speed while improving reliability.
Data Source
AI summary
Embodiments described herein provide a document summarization framework that employs an ensemble of summarization models, each of which is a modified version of a base summarization model to control hallucination. For example, a base summarization model may first be trained on a full training data set. The trained base summarization model is then fine-tuned using a first filtered subset of the training data which contains noisy data, resulting in an “anti-expert” model. The parameters of the anti-expert model are subtracted from the parameters of the trained base model to produce a final summarization model which yields robust factual performance.