Term Encoder Model for Out-of-Vocabulary Embedding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing NLP systems face challenges with out-of-vocabulary (OOV) terms, as static embedding methods cannot convert previously unseen terms into embeddings, and contextual embedding methods do not provide satisfactory representations for named entities and technical terms, especially in texts with many named entities or technical terms.
Innovation Solution
A computer-implemented method is developed to train a term encoder model that generates embeddings and predicts confidence scores by learning from numerical representations of word subunits, allowing dynamic embedding of previously unseen terms and updating embedding matrices efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If static embedding methods are used, then embedding matrices can be efficiently stored and accessed, but out-of-vocabulary terms cannot be converted into embeddings
Solution Approach 1:
The patent segments terms into character n-grams (subunits), allowing the model to process previously unseen terms by composing embeddings from their constituent character sequences. This segmentation enables handling of out-of-vocabulary terms without requiring a complete embedding matrix for all possible terms.
Solution Approach 2:
The patent introduces character n-gram embeddings as an intermediary representation layer between input terms and final term embeddings. This intermediary allows the system to bridge the gap between static pre-trained embeddings and dynamic generation of embeddings for unseen terms.
2Measurement precision
If contextual embedding methods are used, then representations for text not seen at training time can be provided, but satisfactory quality of representation for named entities and technical terms cannot be achieved
Solution Approach 1:
The patent performs preliminary training of the model on pre-existing high-quality embeddings (such as from word2vec or GloVe) to learn the semantic structure and relationships. This preliminary action allows the model to generate quality embeddings for named entities and technical terms while maintaining fast inference speed, avoiding the need for slow contextual embedding methods.
3Measurement precision
If large pre-trained embedding matrices are used, then high-quality embeddings for multiword expressions can be created, but the matrices reach tens of gigabytes in size
Solution Approach 1:
The patent segments terms into character n-grams, reducing the vocabulary space from millions of complete terms to a manageable set of character sequences. This segmentation enables the model to generate embeddings for multiword expressions by composing character-level representations, dramatically reducing the required embedding matrix size from tens of gigabytes to a much smaller footprint.
Solution Approach 2:
The patent changes the parameterization approach from storing complete term embeddings to storing character n-gram embeddings and using a neural network to generate term embeddings on-the-fly. This parameter change transforms the system from a lookup-based approach to a generative approach, reducing storage requirements while maintaining embedding quality.
4Adaptability or versatility
If sentence encoders are used, then dense representations for text not seen at training time can be extracted, but inference latency increases
Solution Approach 1:
The patent extracts and leverages the strengths of pre-trained static embeddings (semantic knowledge and relationships) while removing the computational overhead of full sentence encoder architectures. By using a simpler model that generates embeddings from character n-grams rather than processing entire sentences through complex neural networks, the system achieves fast inference latency while maintaining adaptability to unseen terms.
Data Source
AI summary
According to one embodiment, a method and computer program product for generating a model including a term encoder is provided. The embodiment may include training the model on a training dataset that associates training terms with first embeddings of the training terms. The training includes generating, with the term encoder, second embeddings from numerical representations of word subunits of the training terms with an objective of minimizing distances between the first embeddings and the second embeddings. The word subunits form part of a predetermined set of word subunits. The training includes predicting confidence scores based on the minimized distances. The embodiment may include deploying the model as part of an executable algorithm to allow a user to infer third embeddings and corresponding confidence scores from any input terms written based on word subunits of the predetermined set.