Transformer Summarization via Self-Supervised Control Codes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unsupervised abstractive summarization methods face challenges such as 'hallucinations' due to averaging contradictory input document representations, and existing control methods rely on predefined control tokens with limited domain coverage, leading to low fluency and coherence in generated summaries.
Innovation Solution
A self-supervised method generates enriched training data for a multi-source transformer neural network by creating training points with maximum similarity between input text subsets and reference texts, using classifiers to select control codes that steer the output towards coherent summaries, avoiding hallucinations and improving domain coverage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If unsupervised abstractive summarization methods average representations of input documents, then the summarization process can handle diverse input documents, but this leads to generation of hallucinations due to contradictory representations
Solution Approach 1:
The patent applies preliminary action by performing clustering on input documents before averaging their representations. This pre-processing step groups similar documents together, ensuring that only consistent representations are averaged, thereby preventing hallucinations while maintaining the ability to handle diverse inputs through the clustering mechanism
2Ease of operation
If control codes are applied at inference time using weighted decoding, then the generation process can be controlled, but this results in low fluency and coherence
Solution Approach 1:
The patent applies preliminary action by training the model with control codes incorporated in the training data itself, rather than applying control at inference time. This allows the model to learn proper control mechanisms during training, resulting in better fluency and coherence while maintaining controllability through the learned control code representations
3Ease of operation
If predefined control tokens are used for controlling text generation, then the control mechanism can be implemented, but this implies low domain coverage
Solution Approach 1:
The patent applies universality by using metadata from the input documents themselves as control codes, rather than relying on predefined control tokens. This approach allows the same mechanism to work across diverse domains since the control information is derived from the document content itself, providing both implementability and broad domain coverage
Data Source
AI summary
A method for generating enriched training data for a multi-source transformer neural network for generation of a summary of one or more passages of input text comprises creating, from a plurality of input text sets, training points each comprising an input text subset of the input text set and a corresponding reference input text from the input text set, wherein the size of the input text subset is a predetermined number. Control codes are selected based on reference features corresponding to categorical labels of reference texts in the created training points. The input text is enriched with the selected control codes to generate enriched training data.