Text-to-Text Domain Transfer Using Unlabeled In-Domain Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pre-trained language models face challenges in applying to specialized domains due to the scarcity and high cost of in-domain task labels, and domain transfer performance is limited by the domain gap when using general-domain labels.
Innovation Solution
A text-to-text model is trained using in-domain unlabeled text and general-domain labeled task data through masked language modeling, natural language generation, and understanding, leveraging a multitasking strategy to acquire domain-specific knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If domain-specific pre-training is performed using in-domain task labels, then the model acquires domain-specific knowledge, but the cost and time for acquiring labels increases significantly
Solution Approach 1:
The patent performs preliminary domain adaptation training using only unlabeled in-domain text data before fine-tuning with task labels. This preliminary action of adapting the model to the target domain's language patterns and terminology without requiring labels reduces the subsequent need for extensive labeled data and accelerates the overall development process
Solution Approach 2:
The model performs self-supervised learning through masked language modeling on unlabeled in-domain text, where it learns to predict masked tokens using contextual information. This self-service mechanism allows the model to acquire domain-specific knowledge autonomously without human annotation, eliminating the time-consuming label acquisition process
2Productivity
If general-domain task labels are used for training, then the model can be trained faster, but the transfer performance to specialized domains is limited due to domain gap
Solution Approach 1:
The training process is segmented into distinct stages: first, domain adaptation training on unlabeled in-domain text to bridge the domain gap; second, task-specific fine-tuning on labeled data. This segmentation allows the model to separately learn domain characteristics and task knowledge, improving transfer performance while maintaining training efficiency
Solution Approach 2:
The patent applies preliminary domain adaptation training using in-domain unlabeled text before performing task-specific fine-tuning. This preliminary action adapts the general-domain model to the target domain's language patterns, reducing the domain gap and improving transfer performance while still utilizing efficiently the unlabeled data that can be processed quickly
Data Source
AI summary
Example solutions for zero-shot domain transfer with a text-to-text model train a text-to-text model for a target domain using unlabeled in-domain text training data, and concurrently train the model using labeled general-domain task training data. The in-domain training comprises masked language modeling (MLM) training, and the task training comprises both natural language generation (NLG) training and natural language understanding (NLU) training. The NLG training comprises natural language inference (NLI) training and the NLU training comprises summarization training. The trained model acquires domain-specific task competency, sufficient to perform a language task within the target domain. Suitable target domains include radiology, biomedical, and other medical, legal, and scientific domains. This approach leverages large volumes of general-domain task training data and plentiful unlabeled in-domain text, even as labeled in-domain training data may be unavailable or prohibitively expensive for certain specialized domains.