Globally Normalized Reader for Question Answering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current extractive question answering systems face limitations such as equal computation allocation across entire documents, reliance on expensive bidirectional attention mechanisms, and lack of training data that improves performance or explicitly uses question nature and entity types.
Innovation Solution
The Globally Normalized Reader (GNR) model approaches question answering as a factorized search process, using conditional computation and beam search to allocate resources efficiently, and Type Swaps data-augmentation strategy aligns named entities with knowledge base entities to enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If bidirectional attention mechanisms are used to improve question answering accuracy, then measurement precision is improved, but use of energy increases
Solution Approach 1:
The patent segments the document processing into two distinct phases: (1) generating candidate answer spans using efficient unidirectional attention, and (2) ranking these candidates using bidirectional attention. This segmentation allows the expensive bidirectional attention to be applied only to a small subset of candidate spans rather than the entire document, thereby reducing computational cost while maintaining accuracy.
Solution Approach 2:
The patent applies different quality levels of attention mechanisms to different parts of the processing pipeline. Unidirectional attention (lower computational cost) is used for initial candidate generation across the entire document, while bidirectional attention (higher computational cost) is applied locally only to rank the generated candidate spans. This local quality approach optimizes the trade-off between accuracy and computational expense.
2Measurement precision
If computation is allocated equally across entire documents to improve measurement precision, then question answering accuracy is improved, but productivity decreases
Solution Approach 1:
The patent divides the computation into two stages: first generating candidate spans with efficient unidirectional attention, then ranking only these candidates with bidirectional attention. This segmentation enables the system to process longer documents faster by avoiding exhaustive bidirectional attention across the entire document, thus improving productivity while maintaining accuracy through the two-stage approach.
Solution Approach 2:
The patent performs partial action by generating a limited set of candidate answer spans rather than evaluating all possible spans. This partial approach (considering only top candidates) reduces the computational burden significantly while still achieving high accuracy, as the most likely answers are identified and ranked without exhaustive search.
3Measurement precision
If data augmentation is applied to improve measurement precision, then training data quality is improved, but device complexity increases
Solution Approach 1:
The patent uses copying by creating augmented training examples through paraphrasing existing question-answer pairs. Instead of collecting new data, the system generates synthetic training data by copying and transforming existing examples, thereby improving training data quality without adding physical or operational complexity to the system.
Solution Approach 2:
The patent applies parameter changes by transforming existing training data through various operations such as paraphrasing questions, replacing entities with synonyms, and modifying answer spans. These parameter transformations create diverse training examples from a single source, improving data quality while using simple text processing operations rather than complex system modifications.
Data Source
AI summary
Presented herein are systems and methods for question answering (QA). In embodiments, extractive question answering (QA) is cast as an iterative search problem through the document's structure: select the answer's sentence, start word, and end word. This representation reduces the space of each search step and allows computation to be conditionally allocated to promising search paths. In embodiments, globally normalizing the decision process and back-propagating through beam search makes this representation viable and learning efficient. Various model embodiments, referred to as Globally Normalized Readers (GNR), achieve excellent performance. Also introduced are embodiments of data-augmentation to produce semantically valid examples by aligning named entities to a knowledge base and performing swaps new entities of the same type. This methodology also improved the performance of GNR models and is of independent interest for a variety of natural language processing (NLP) tasks.