Hybrid Text Summarization for Long Documents and Low Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automatic summarization techniques struggle to handle large volumes of text data with diverse contexts, leading to inefficiencies and unsatisfactory outputs due to resource constraints and token limits, while extractive methods often produce redundant summaries.
Innovation Solution
A hybrid summarization model combining abstractive and extractive methods to identify salient sentences and generate concise summaries, using a combination of extractive methods to filter important sentences and abstractive methods to rewrite them, leveraging models like MPNet and K-means clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional automatic summarization techniques are used to process large volumes of text data from diverse contexts, then processing speed and scalability are improved, but the quality and accuracy of summaries deteriorate due to resource constraints and token limits
Solution Approach 1:
The patent divides the text processing task into multiple stages: first segmenting text into sentences, then into phrases, and finally into keywords. This hierarchical segmentation allows the system to process large volumes of text efficiently while maintaining summary quality by focusing computational resources on extracting and synthesizing key information at each level rather than processing entire documents as single units.
Solution Approach 2:
The patent introduces an intermediary extraction layer that identifies and isolates key phrases and concepts from the source text before generating summaries. This intermediary step acts as a mediator between the raw text input and the final summary output, filtering out redundant information and preserving only the most salient points, thereby maintaining high summary quality even when processing large datasets with resource constraints.
2Productivity
If extractive summarization methods are used to quickly generate summaries, then processing efficiency is improved, but redundancy in summaries increases
Solution Approach 1:
The patent applies extraction by identifying and removing redundant information from the source text before generating summaries. The system extracts only the most important phrases, concepts, and key points while discarding repetitive or less significant content. This extraction process ensures that the resulting summaries are concise and non-redundant while still capturing the essential information efficiently.
Solution Approach 2:
The patent changes parameters by dynamically adjusting the level of extraction and summarization based on the characteristics of the input text. The system monitors metrics such as phrase frequency, information density, and semantic similarity to determine how much content to extract and how to phrase the summary, thereby optimizing the balance between processing efficiency and summary quality by adapting to different text types and contexts.
3Manufacturing precision
If abstractive summarization is applied to all text clusters, then summary quality is improved, but computational resources and time are excessively consumed
Solution Approach 1:
The patent applies partial action by using abstractive summarization selectively rather than universally. The system first applies extractive methods to generate initial summaries for all text clusters, then applies abstractive summarization only to specific clusters where the extractive summaries are deemed insufficient based on quality metrics. This partial application of the more resource-intensive abstractive method maintains high summary quality where needed while avoiding unnecessary computational overhead for cases where extractive methods suffice.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods implement a summarization model that includes a combination of abstractive and extractive summarization are provided. Extractive summarization is applied to identify the most salient sentences from a text corpus. Abstractive summarization is utilized to generate a topic summary based on filtered outputs from the extractive summarization.