Language Model Prompt Adaptation With Domain Keyword Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pre-trained language models struggle to perform effectively in domain-specific settings due to the lack of tailored domain-specific data for pre-training, necessitating the use of domain-specific models that are hard to obtain.
Innovation Solution
A method to dynamically adapt general-domain pre-trained language models by incorporating a tokenizer and transformer to create a soft prompt with domain-specific keywords, allowing effective use in specific domains without requiring additional pre-trained models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If domain-specific pre-trained language models are used, then performance on domain-specific tasks is improved, but resource consumption and storage requirements increase
Solution Approach 1:
The patent embeds domain-specific keyword embeddings within the existing general-domain language model architecture. Instead of creating a separate domain-specific model, it nests the domain knowledge (keyword embeddings) inside the general model's tokenization and embedding layers, allowing the general model to be adapted to specific domains without increasing overall storage requirements
Solution Approach 2:
The patent makes a single general-domain language model capable of performing multiple domain-specific tasks by dynamically injecting domain-specific keyword embeddings. The same base model serves universal purposes across different domains by adjusting which keywords are embedded, eliminating the need for separate domain-specific models
2Reliability
If domain-specific pre-trained language models are trained, then domain-specific performance is improved, but data requirements and training resources increase
Solution Approach 1:
The patent pre-computes and stores embeddings for domain-specific keywords offline before deployment. This preliminary action allows the system to quickly adapt to new domains by simply selecting and injecting appropriate pre-computed keyword embeddings, without requiring extensive domain-specific training data or retraining the entire language model
Solution Approach 2:
The patent extracts only the essential domain-specific information (keywords and their embeddings) from domain-specific data, rather than using the entire domain corpus for training. This extraction approach significantly reduces data requirements while still capturing the essential domain knowledge needed for effective performance
3Adaptability or versatility
If hard prompts or soft prompts are used, then adaptability to different domains is improved, but they are not optimized for domain-specific settings
Solution Approach 1:
The patent applies local quality by injecting domain-specific keyword embeddings at specific locations within the model architecture (in the tokenization and embedding layers) rather than uniformly modifying the entire model. This localized modification allows the model to maintain its general capabilities while being optimized for specific domains where the keywords are relevant
Solution Approach 2:
The patent creates a composite approach by combining general-domain pre-trained model weights with domain-specific keyword embeddings. This composite structure leverages the strengths of both general-domain training (broad language understanding) and domain-specific knowledge (relevant terminology and concepts), achieving both adaptability and domain-specific optimization
Data Source
AI summary
A device and computer implemented method for operating a knowledge base including a language model. The language model is pre-trained for a first domain and comprises a tokenizer and a transformer. The method includes providing an input sentence of a second domain, tokenizing the input sentence with the tokenizer into tokens, transforming the tokens with the transformer to a sentence embedding, providing a first set of embeddings that represent a prompt of the first domain, providing a second set of embeddings, wherein the second set of embeddings comprises the first set of embeddings and a set of embeddings that represent keywords of the second domain, determining a prompt depending on the first set of embeddings and the second set of embeddings, transforming at least one of the tokens and the prompt with the transformer to an output of the knowledge base.