Meta-Learning Task Model for Few-Shot Text Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for natural language processing face challenges in adapting large-scale language models to new domains with limited label data, particularly in few-shot text classification, where securing sufficient label data is costly and time-consuming, and pre-trained models like BERT and GPT-3 struggle to perform effectively.
Innovation Solution
A meta-learning based method that calculates task-adaptation and meta-optimization losses to update a task model, generating well-adapted text embeddings for new domains using a computing device, which includes inputting few-shot text data into a language model, generating token and class embeddings, and synthesizing feature information through relation and gating networks to produce domain-specific text embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transfer learning is performed by tuning a large-scale language model with sufficient label data from a target domain, then the model can be well-adapted to the target domain, but securing sufficient label data is costly and time-consuming
Solution Approach 1:
The patent applies preliminary action by pre-training the language model on large-scale general domain data before deployment. This preliminary training endows the model with general language understanding capabilities, eliminating the need for extensive target domain label data collection while maintaining good adaptation performance.
Solution Approach 2:
The patent achieves universality by designing a language model that can perform multiple tasks (text classification, question answering, summarization) across different domains without requiring domain-specific tuning. The model uses universal attention mechanisms and transformer architecture that generalize across diverse NLP tasks and domains.
2Ease of manufacture
If a small amount of label data is used for tuning, then data collection cost is reduced, but the language model cannot be adequately tuned to the target domain
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting attention weights, learning rates, and model configuration parameters based on the available data amount and task requirements. This allows the model to achieve good adaptation performance even with limited label data by optimizing parameters rather than relying on extensive data.
Solution Approach 2:
The patent uses copying by leveraging patterns and representations learned from general domain data and applying them to target domain tasks. The model copies useful linguistic patterns, syntax structures, and semantic relationships from pre-training data, reducing dependence on target domain label data.
3Productivity
If text embeddings are generated by a pre-trained language model for classification, then classification can be performed, but the model cannot be properly tuned to new domains or label categories without sufficient data
Solution Approach 1:
The patent applies dynamics by implementing dynamic attention mechanisms that can adaptively adjust to new domains and tasks. The attention weights are dynamically computed based on the input sequence and task requirements, allowing the model to flexibly adapt to new domains without retraining, maintaining both speed and adaptability.
Solution Approach 2:
The patent uses an intermediary layer between the pre-trained language model and the classification task. This intermediary includes task-specific attention heads and projection layers that adapt the general language representations to specific classification tasks, enabling fast classification while maintaining domain adaptability.
Data Source
AI summary
The present disclosure according to at least one embodiment provides a method performed by a computing device for generating a task model based on meta-learning, including calculating a task-adaptation loss of the task model, the calculating the task-adaptation loss being based on a result of training the task model by using a training data set, updating the task model based on the task-adaptation loss, calculating a meta-optimization loss of the updated task model by using a validation data set, and further updating the updated task model based on the meta-optimization loss.