Field-Aware Language Model Training for Short-Sentence Paraphrases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional general-purpose large-scale language models face difficulties in learning similarity of paraphrase expressions in short-sentence documents, such as infrastructure-domain trouble reports, due to limited context information, leading to distant embedded expressions for words with the same meaning.
Innovation Solution
An information learning apparatus that includes a field information acquiring unit, a field loss calculating unit, and a model update unit to utilize field information and calculate losses, updating model parameters to bring similar paraphrase expressions closer in the embedding space by considering semantic fields and latent relations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pre-training is executed using a large-scale corpus of short-sentence documents, then the model can be trained on available infrastructure-domain data, but the model fails to learn similarity of paraphrase expressions due to limited context information
Solution Approach 1:
The patent introduces field information (metadata such as trouble type, component, phenomenon) as an intermediary element that mediates between the limited short-sentence context and the semantic meaning of words. This field information provides additional contextual cues that enable the model to learn paraphrase similarity despite the brevity of the original documents
Solution Approach 2:
The patent adds a new dimension to the training data by incorporating field information alongside the text data. This creates a multi-dimensional input space (text + field metadata) that compensates for the limited contextual information in short sentences, enabling better learning of word embeddings and paraphrase relationships
2Ease of manufacture
If conventional pre-training methods are used on short-sentence documents, then training can be completed with standard approaches, but the embedded expressions of words with the same meaning become distant in the vector space
Solution Approach 1:
The patent modifies the training parameters by introducing field information as an additional input feature and changing the loss function to incorporate field-aware components. This parameter change enables the model to produce higher quality word embeddings that correctly group paraphrases together, while maintaining relative simplicity in the training implementation
Data Source
AI summary
An information learning apparatus includes a processing circuitry. The processing circuitry is configured to: acquire text information and field information included in the text information; calculate a field loss based on the text information and the field information; and store parameters of a trained model, and updating the parameters of the model based on the field loss.