Shared Pre-Trained Transformer for Joint NER, RE, and AD Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training machine learning models to perform named entity recognition (NER), relation extraction (RE), and assertion detection (AD) are inefficient in terms of time and resource consumption, as they typically involve separate models for each task, and multi-task learning frameworks do not effectively incorporate AD.
Innovation Solution
A multi-task learning framework using a shared pre-trained language model with a transformer architecture, where NER, RE, and AD tasks are jointly trained with shared representations and balanced gradients, allowing for efficient and simultaneous learning across all tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate models are used for each NLP task (NER, RE, AD), then each model can be optimized for its specific task, but time consumption and resource usage increase significantly
Solution Approach 1:
The patent combines three separate NLP tasks (Named Entity Recognition, Relation Extraction, and Assertion Detection) into a single unified model that processes all tasks simultaneously. This merging approach reduces the number of separate models needed from three to one, thereby decreasing training time and computational resource consumption while maintaining task-specific performance through dedicated output layers for each task.
Solution Approach 2:
The unified model is designed with multi-functionality to perform NER, RE, and AD tasks using a shared pre-trained language model architecture. The model incorporates task-specific modules that build upon the shared representations, allowing a single model to serve multiple NLP functions efficiently, thus reducing overall system complexity and resource requirements compared to separate specialized models.
2Productivity
If multi-task learning framework is used to train unified model, then training efficiency improves, but the framework complexity increases
Solution Approach 1:
The unified model is segmented into distinct functional modules: a shared pre-trained language model base and task-specific output modules for NER, RE, and AD. This segmentation allows each task to be trained independently within the unified framework while maintaining shared representations, thereby managing framework complexity through modular architecture while achieving improved training efficiency through multi-task learning.
Solution Approach 2:
The framework applies local quality by providing task-specific processing modules at the output layer while maintaining a shared representation layer. Each task (NER, RE, AD) has its own dedicated modules that process information specifically for that task, while the shared layers provide common linguistic understanding, thus balancing framework complexity with training efficiency through localized task optimization.
3Adaptability or versatility
If assertion detection is added to existing NER and RE frameworks, then functional versatility improves, but the difficulty of integrating and training the model increases
Solution Approach 1:
The framework achieves universality by integrating three NLP functions (NER, RE, and AD) into a single unified model that shares common representations. This multi-functional design allows the model to handle diverse NLP tasks simultaneously, improving functional versatility while managing integration difficulty through a coherent shared architecture that leverages common linguistic features across all tasks.
Solution Approach 2:
The framework employs preliminary action by using a pre-trained language model as the foundation for the unified architecture. This pre-trained model provides pre-learned linguistic representations that simplify the integration of new tasks (NER, RE, AD), reducing the complexity of training and allowing the framework to handle assertion detection and other tasks more efficiently by building upon existing knowledge rather than training from scratch.
Data Source
AI summary
Disclosed are machine learning techniques directed to training a machine learning model for the combined learning of multiple natural language processing (NLP) tasks. The NLP tasks may be named entity recognition (NER), relation extraction (RE), and assertion detection (AD) tasks. The machine learning model may be a multi-layer transformer model. Training the machine learning model may involve first training the NER module on the NER task, and thereafter training the RE module on the RE task while the AD module is simultaneously trained on the AD task. Training the machine learning model may alternatively involve training the NER module on the NER task concurrently with training the RE module on the RE task and training the AD module on the AD task. The trained machine learning model can predict entities and entity types in newly provided text, along with relations between the entities and assertions associated with the entities.