Machine Reading Comprehension Model Training with Distance-Based Label Smoothing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine reading comprehension technologies face challenges in achieving high accuracy due to reliance on large-scale pre-trained language models that struggle with understanding text context, often requiring manually constructed datasets and simple questions, leading to low answer accuracy.
Innovation Solution
A method and apparatus for training a machine reading comprehension model that calculates the distance between words and answer labels, using a smooth function to generate probability values, which are then used as smoothed labels for model training, incorporating stop words near answer boundaries to improve prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large-scale pre-trained language models are used for machine reading comprehension, then the model can process complex text, but the training time increases and answer accuracy does not improve significantly
Solution Approach 1:
The patent extracts and utilizes stop words from the text corpus, which are typically discarded in traditional NLP approaches. By identifying stop words that appear near answer labels and incorporating them into the training data with assigned probability values, the method extracts useful information from previously ignored elements, improving answer accuracy without requiring larger models or longer training times
Solution Approach 2:
The patent changes the parameter representation by introducing probability values for stop words based on their distance from answer labels. Instead of treating all words uniformly or ignoring stop words, the method assigns continuous probability values (e.g., using a smooth function like softmax) that reflect the likelihood of each stop word being part of or near the answer, enabling more nuanced training signals
2Ease of manufacture
If manually constructed datasets with simple questions are used, then the training process is simpler, but the answer accuracy remains low
Solution Approach 1:
The patent introduces stop words as intermediary elements that bridge the gap between simple manually constructed datasets and accurate answer prediction. Stop words serve as mediators that provide additional contextual information about answer boundaries and relevance, enhancing the information content of simple datasets without requiring complex manual construction or annotation
Data Source
AI summary
Disclosed is an apparatus for training a machine reading comprehension model. The apparatus is inclusive of a distance calculation part configured to calculate, based on a position of each word within a training text and a position of an answer label within the training text, a distance between the same word and the answer label; a label smoothing part configured to input the distance between the same word and the answer label into a smooth function to obtain a probability value corresponding to the same word, outputted from the smooth function; and a model training part configured to make the probability value corresponding to the same word serve as a smoothed label of the same word so as to train the machine reading comprehension model.