Abstractive Document Summarization Using Extractive Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning algorithms, particularly recurrent neural networks (RNNs), face limitations in optimizing and processing long sequences and parallelization, making them inefficient for generating summaries of documents with thousands of words, and lack the fluency and coherence of human-generated summaries in abstractive summarization.
Innovation Solution
The approach involves a two-part summarization process: an extractive step using hierarchical document models, such as pointer networks and sentence classifiers, to condition transformer language models on relevant information, improving the generation of abstractive summaries by splitting the task into sentence extraction and summary generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If recurrent neural networks (RNNs) are used for sequence processing, then the model can capture sequential dependencies, but the processing efficiency and parallelization capability deteriorate
Solution Approach 1:
The patent segments the sequential processing task into parallel attention computations. Instead of processing sequences step-by-step through RNN layers, the encoder and decoder simultaneously compute attention weights for all position pairs, dividing the computational workload into independent parallel operations that can be executed concurrently on GPU hardware.
Solution Approach 2:
The patent replaces the mechanical sequential computation mechanism of RNNs with an attention-based parallel computation system. The attention mechanism uses matrix operations to compute relationships between all sequence elements simultaneously, substituting the stepwise mechanical processing of RNNs with parallel linear algebra operations that leverage modern hardware capabilities.
2Measurement precision
If extractive summarization is used, then the most relevant information is identified, but the fluency and coherence of the summary deteriorates
Solution Approach 1:
The patent introduces an attention mechanism as an intermediary between the extractive selection process and the final summary generation. The attention weights serve as a mediator that selectively emphasizes relevant source positions while maintaining the ability to generate fluent, coherent output sequences that go beyond simple extraction, bridging the gap between precision and fluency.
Solution Approach 2:
The patent merges extractive summarization and abstractive summarization capabilities into a single unified model. The encoder-decoder architecture with attention combines the strength of extractive methods (identifying relevant information) with the strength of abstractive methods (generating fluent text), allowing the model to both select key information and reformulate it coherently.
3Quantity of substance
If the document length increases, then more information is available for summarization, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-computing position embeddings and encoding the entire input document into contextual representations before the decoding phase. The encoder processes all input tokens and generates hidden states in advance, creating a compressed representation that can be efficiently queried during decoding, reducing the computational burden during the actual summarization generation.
Solution Approach 2:
The patent transforms the one-dimensional sequential processing problem into a two-dimensional attention matrix problem. By computing attention weights across all position pairs simultaneously in a matrix operation, the model changes the computational dimension from sequential steps to parallel matrix operations, improving efficiency for long sequences through better hardware utilization.
Data Source
AI summary
There is provided a non-transitory storage medium and a system for generating an abstractive summary of a document using an abstractive machine learning algorithm (MLA). A document including a plurality of text sequences is received. An extractive summary of the document is generated, the extractive summary including a set of summary text sequences which is a subset of the plurality of text sequences. The abstractive MLA generates, based on the set of summary text sequences and at least a portion of the plurality of text sequences, an abstractive summary of the document including a set of abstractive text sequences, at least one abstractive text sequence not being included in the plurality of text sequences.