Language Model Updating with Domain Word Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models specialized for specialized domains require extensive learning periods and do not achieve performance comparable to pre-learned models, despite the availability of pre-learned models for general domains.
Innovation Solution
A learning apparatus that updates language model parameters using a conversion unit and a parameter update unit, incorporating word vectors learned from a specialized domain corpus to enhance performance without significantly increasing the learning period.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-learning 1 is performed using a corpus specialized in a specialized domain, then the language model achieves high performance in that domain, but the learning period becomes enormously long
Solution Approach 1:
The patent applies preliminary action by pre-learning word vectors from general domain corpus before performing domain-specific pre-learning 1. This preliminary preparation of word vectors accelerates the subsequent specialized domain learning process, reducing the enormously long learning period while maintaining high model performance in specialized domains like medical care and law
2Adaptability or versatility
If pre-learning 2 is performed using training data of target task, then the model adapts to the target task, but the performance is still insufficient compared to pre-learned models in specialized domains
Solution Approach 1:
The patent performs pre-learning 2 as a preliminary action before Fine-Tuning, using both general domain corpus and specialized domain corpus. This intermediate pre-learning step prepares the model with domain-specific knowledge and word vectors, enabling it to achieve sufficient performance in target tasks while adapting effectively, bridging the gap between general pre-trained models and specialized domain models
Data Source
AI summary
A learning apparatus includes a memory and at least one processor connected to the memory, wherein the processor configured to: convert input text data into a feature amount sequence based on a language model; and update parameters of the language model based on the text data, the feature amount sequence, and a word vector learned in advance.