Hierarchical Multi-Task Term Embedding for Synonym Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing synonym prediction methods face challenges in learning effective representations for entities and descriptions, particularly in domains with high language variability and low knowledge base coverage, such as Chinese medical text, where existing methods ignore rich relational information among entities and require large amounts of labeled data not readily available.
Innovation Solution
A hierarchical multi-task term embedding model is introduced, which incorporates semantic type knowledge and relational information among entities through an auxiliary task of neighboring word semantic type prediction, enhancing the skip-gram model to learn more representative embeddings and improve synonym prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If word embedding methods are used to learn distributed vector representation from large corpus, then representation quality improves, but the model ignores rich relational information among entities
Solution Approach 1:
The patent combines word embedding methods with knowledge base relational information by integrating entity embeddings that capture both distributional semantics from corpus and structured relational knowledge from knowledge base, thereby preserving relational information while maintaining representation quality
Solution Approach 2:
The patent introduces an auxiliary task of neighboring word semantic type prediction as an intermediary mechanism that bridges the gap between distributional word embeddings and structured knowledge base relations, allowing the model to learn and preserve relational information through the auxiliary task
2Measurement precision
If complex models with auxiliary tasks are introduced to leverage semantic knowledge, then synonym prediction accuracy improves, but the amount of labeled data required increases
Solution Approach 1:
The patent enables the model to leverage existing knowledge base structures and semantic type information without requiring additional labeled synonym data, allowing the system to serve itself by utilizing already-available structured knowledge to improve prediction accuracy
Solution Approach 2:
The patent performs preliminary action by pre-structuring knowledge base entities with semantic types and relations before the synonym prediction task, so that the auxiliary task can leverage this pre-organized information without requiring labeled training data for each specific prediction task
3Adaptability or versatility
If knowledge bases are expanded to cover more informal language variations, then coverage improves, but the difficulty of constructing labeled data increases
Solution Approach 1:
The patent replaces the mechanical process of manually constructing labeled data for each language variation with an automated embedding-based system that can generalize across informal language variations through distributional semantics and transfer learning from the auxiliary task
Data Source
AI summary
Due to the high language use variability in real-life, manual construction of semantic resources to cover all synonyms is prohibitively expensive and may result in limited coverage. Described herein are systems and methods that automate the process of synonymy resource development, including both formal entities and noisy descriptions from end-users. Embodiments of a multi-task model with hierarchical task relationship are presented that learn more representative entity/term embeddings and apply them to synonym prediction. In model embodiments, a skip-gram word embedding model is extended by introducing an auxiliary task “neighboring word/term semantic type prediction” and hierarchically organize them based on the task complexity. In one or more embodiments, existing term-term synonymous knowledge is integrated into the word embedding learning framework. Embeddings trained from the multi-task model embodiments yield significant improvement for entity semantic relatedness evaluation, neighboring word/term semantic type prediction, and synonym prediction compared with baselines.