Automated Abstractive Summarizer Selection via Extractive Reference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text summarization technologies lack an efficient method for automatically evaluating and selecting the best summarizer from a group of candidates, leading to suboptimal summary generation and requiring human intervention.
Innovation Solution
A system that compares the performance of candidate abstractive summarizers to a base summarizer using metrics like Rouge-L f-measure, automatically identifying the best summarizer for a document type by generating summaries and evaluating their similarity to a reference extractive summary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple candidate summarizers are evaluated and compared to select the best one, then summary quality is improved, but the complexity of the system increases
Solution Approach 1:
The patent introduces an extractive summarizer as an intermediary reference system to evaluate and compare multiple candidate abstractive summarizers. This mediator generates reference summaries that serve as the ground truth for evaluation, enabling automated selection of the best performing summarizer without requiring manual assessment of each candidate's output quality
Solution Approach 2:
The system performs self-evaluation by automatically comparing candidate summarizer outputs against the reference extractive summary using evaluation metrics. The best summarizer is selected through automated performance measurement rather than human intervention, allowing the system to self-optimize for different document types
2Extent of automation
If automated evaluation and selection of summarizers is implemented, then human intervention is reduced, but the measurement and detection difficulty increases
Solution Approach 1:
The patent replaces manual human evaluation of summarizer quality with automated computational metrics. Instead of requiring human reviewers to assess summary accuracy and quality, the system uses automated comparison algorithms that measure similarity between candidate summaries and reference summaries, substituting mechanical computation for human judgment
Solution Approach 2:
The extractive summarizer serves as an intermediary reference that enables automated measurement. By generating objective reference summaries that can be systematically compared against candidate outputs using standard evaluation metrics, the system transforms the subjective task of quality assessment into an objective automated measurement process
3Measurement precision
If the best summarizer is selected for each document type, then summary accuracy is improved, but the time required for evaluation increases
Solution Approach 1:
The system performs preliminary evaluation of multiple candidate summarizers across different document types in advance. By pre-identifying the best performing summarizer for each document type through automated evaluation, the system establishes a lookup table of optimal summarizers that can be quickly selected during actual summarization tasks without repeating the full evaluation process
Solution Approach 2:
The patent segments the evaluation process by document type, evaluating and selecting summarizers separately for different categories such as news articles, research papers, and financial documents. This segmentation allows the system to optimize accuracy for each specific document type while managing evaluation time by focusing comparisons within defined categories rather than across all possible document types
Data Source
AI summary
Content to be summarized is received and analyzed using an extractive summarizer to determine a reference extractive summary of the content. The content is further analyzed using a plurality of different abstractive summarizers to determine candidate abstractive summaries of the content. Each of the candidate abstractive summaries is compared with the reference extractive summary to determine corresponding evaluation metrics. Based at least in part on the evaluation metrics, one of the candidate abstractive summaries is selected as a selected summary to be provided.