Consistency Evaluation for Document Summaries Using Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative summarization models often produce factually inconsistent summaries due to their ability to rephrase content using different language, which is problematic in domains requiring factual consistency, and existing evaluation methods, such as Natural Language Inference models and large language models, are either ineffective or computationally expensive.
Innovation Solution
A system generates a dataset with annotated model-generated summaries using a language model neural network, allowing for the training of a student consistency evaluation model that evaluates summary consistency without requiring rule-based perturbations or human-written summaries, and is computationally efficient.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generative summarization models are used to create abstractive summaries, then the summaries become more informative and interesting, but factual consistency deteriorates
Solution Approach 1:
The patent introduces an intermediary consistency evaluation model that acts as a mediator between the generative summarization model and the final output. This model evaluates whether the generated summary is consistent with the source document by comparing key information, entities, and relationships, thereby maintaining both the creativity of generative models and the factual accuracy required.
2Ease of operation
If Natural Language Inference models are used for evaluating factual consistency, then evaluation capability is provided, but effectiveness deteriorates for longer summaries
Solution Approach 1:
The patent segments the consistency evaluation process into multiple components: extracting key information from the document, identifying important entities and relationships, comparing these elements against the summary, and evaluating consistency for each aspect separately. This segmentation enables effective evaluation of long summaries by breaking them down into manageable consistency checks rather than attempting to evaluate the entire summary as a single unit.
3Reliability
If large language models are used to evaluate generative tasks, then evaluation power is increased, but computational cost and latency increase excessively
Solution Approach 1:
The patent employs a more computationally efficient consistency evaluation model that does not require the excessive computational resources of large language models. The model uses targeted comparisons of key information, entities, and relationships rather than full-scale language model inference, making it suitable for practical deployment where computational efficiency and low latency are critical while still providing reliable consistency evaluation.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating, using a language model, a data set for use in performing consistency evaluation for document summaries. For example, the data set can be used to train or evaluate a consistency evaluation neural network.