Neural Network Multi-Document Question Answering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine reading models trained on mono-document question answering datasets struggle with multi-document question answering, often providing irrelevant answers due to their training on passages with embedded answers, and face challenges in creating large-scale datasets with accurate answer positions in large corpora, leading to ambiguity and inefficiency.
Innovation Solution
A computer-implemented method using a neural network model trained with distant supervision and distance-based ranking loss to compute scores for text portions, identifying the most relevant answer in a corpus of documents, and selecting the highest scored text as the answer, which does not require extensive labeling of training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine reading models are trained on mono-document question answering datasets, then they can locate answers in given passages, but they always provide an answer even when the passage does not contain the answer
Solution Approach 1:
Instead of training models to always find answers in passages, the patent inverts the approach by training models to recognize when passages do NOT contain answers. The loss function is designed to penalize false positives (claiming an answer exists when it doesn't) rather than focusing only on finding correct answers, thereby improving reliability without sacrificing precision
Solution Approach 2:
The patent implements a feedback mechanism through the loss function that provides signals about whether predicted answer spans are actually correct or hallucinated. The model receives feedback during training about false positives and adjusts its predictions accordingly, learning to distinguish between passages that contain answers and those that don't
2Adaptability or versatility
If a search engine is used to retrieve passages for multi-document question answering, then more potential answers can be found, but the machine reading model always provides an answer even if the passage is irrelevant
Solution Approach 1:
The patent applies the inversion principle by retraining the machine reading model on multi-document data with a focus on identifying non-answer passages. The model learns to distinguish relevant from irrelevant passages across multiple documents, preventing it from generating sensible-sounding but incorrect answers when passages are irrelevant
Solution Approach 2:
The patent changes the training parameters by using a new loss function specifically designed for multi-document question answering. This loss function modifies how the model evaluates its predictions, penalizing false positives more heavily and adjusting the decision threshold for claiming an answer exists, thereby improving reliability in multi-document settings
3Measurement precision
If large-scale multi-document datasets with ground truth answer positions are created, then model training can be improved, but such datasets are difficult to create due to ambiguity
Solution Approach 1:
The patent applies self-service by using the search engine to automatically generate training data with ground truth answer positions. Instead of manual annotation, the system uses the search engine's retrieval results to identify relevant passages and extract answer positions automatically, greatly reducing dataset creation complexity while maintaining precision
Solution Approach 2:
The patent performs preliminary action by pre-processing the training data through search engine retrieval before model training. The search engine identifies relevant passages and establishes ground truth answer positions in advance, creating a ready-to-use training dataset that eliminates the need for complex manual annotation during the model development phase
Data Source
AI summary
A computer implemented method for multi-document question answering is performed on a server communicating with a client device over a network. The method includes receiving a runtime question from a client device and retrieving runtime documents concerning the runtime question using a search engine. Runtime answer samples are identified in the retrieved runtime documents. A neural network model, trained using distant supervision and distance based ranking loss, is used to compute runtime scores from runtime question data representing the runtime question and from a runtime answer sample representing a first portion of text from the corpus of documents, where each runtime score represents a probability an answer to the runtime question is present in the runtime answer samples. A runtime answer is selected from the runtime answer samples corresponding to the highest runtime score sent to the client device.