Sense-Preserving Text Embedding via Ontology-Guided Contrastive Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current textual encoding and embedding methods fail to accurately preserve the meaning and sense of words, leading to poor performance in retrieval tasks due to limitations in capturing context and multiple senses of words, resulting in low precision and recall, especially when encountering out-of-vocabulary words.
Innovation Solution
A novel word embedding method using a contrastive machine learning approach with a labeled dataset from language ontology data structures, incorporating part-of-speech, sense, and synonyms to create similarity lists, which are then used to train an encoder-decoder network for sense and similarity preserving embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional word embedding methods (e.g., Word2Vec, one-hot encoding) are used to represent text, then the text can be converted to numeric form for AI processing, but the meaning and sense of words are not accurately preserved, leading to poor retrieval precision and recall
Solution Approach 1:
The patent extends traditional word embeddings by incorporating sense information as an additional dimension. Each word is represented not just by its vector embedding but by a tuple (word, sense), where sense is identified through ontology data structures. This dimensional extension allows the system to preserve multiple meanings of words while maintaining their vector representations, thereby improving retrieval precision without losing semantic information.
Solution Approach 2:
The patent introduces ontology data structures as an intermediary between raw text and vector embeddings. The ontology provides sense information that mediates the representation process, allowing the system to disambiguate words with multiple meanings. This intermediary layer enables accurate sense identification while preserving the original word embeddings, thus preventing information loss about word meanings.
2Productivity
If traditional word embedding methods are used, then text processing can be performed efficiently, but the system fails to capture context and multiple senses of words, resulting in low precision and recall
Solution Approach 1:
The patent performs preliminary sense identification using ontology data structures before the main text processing task. By pre-identifying the sense of each word through ontology lookups, the system prepares sense-tagged representations that can be directly used in subsequent processing. This preliminary action ensures that multiple senses are captured upfront without significantly increasing processing time, thereby improving reliability while maintaining productivity.
Solution Approach 2:
The patent segments word representation into two distinct components: the word itself (with its vector embedding) and its sense (identified through ontology). This segmentation allows the system to process words efficiently using standard embeddings while simultaneously capturing multiple senses through the separate sense component. The segmented approach enables parallel processing of both aspects, maintaining productivity while improving precision and recall.
3Adaptability or versatility
If standard word embedding techniques are applied, then text can be vectorized for AI mechanisms, but out-of-vocabulary words cannot be handled effectively
Solution Approach 1:
The patent makes the embedding system universal by incorporating ontology-based sense identification that can handle any word, including out-of-vocabulary words. The ontology data structures provide a comprehensive framework that can represent senses for words not present in standard embedding vocabularies. This multi-functional approach allows the system to process both in-vocabulary and out-of-vocabulary words using the same mechanism, thereby improving adaptability without proportionally increasing complexity.
4Loss of information
If simple text encoding is used, then the encoding process is computationally efficient, but the sense attribute of terms is not preserved
Solution Approach 1:
The patent enables the encoding system to serve itself by using ontology data structures that can be queried and processed efficiently. The ontology provides pre-organized sense information that can be retrieved through efficient lookup mechanisms, allowing the system to preserve sense attributes without requiring complex external resources. This self-service approach minimizes additional computational energy while maintaining sense information.
Data Source
AI summary
Mechanisms for implementing a text encoder and text encoder operations are provided. A contrastive machine learning training operation trains an encoder of a machine learning computer model, to learn a sense and similarity preserving embedding that operates to encode input natural language text data to generate encoded natural language text data based on a sense attribute of one or more terms in the input natural language text data. The contrastive machine learning training operation learns to separate positive samples in training data from negative samples in the training data. The trained encoder processes a term specified in an input natural language text to generate an encoded natural language text based on the learned embedding and inputs, to a downstream computing system, the encoded natural language text to cause the downstream computing system to perform a computer natural language processing operation based on the embedding.