Language Model Architecture for Multi-Domain Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional language model training methods, such as full fine-tuning, face issues like data forgetting and excessive resource and time consumption as the number of domains increases, necessitating a more efficient approach to handle multiple domains effectively.
Innovation Solution
The proposed solution involves an electronic device with a language model architecture that includes an input embedding layer, positional encoding layer, sub-network layers with multi-head attention, feed-forward layers, and a softmax layer, allowing for efficient training by branching specific layers and minimizing weight increases, thereby reducing computational resources and training time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If full fine-tuning method is used to train a language model, then the entire weight of the language model is updated, but data forgetting occurs and previously trained data is lost
Solution Approach 1:
The language model weights are segmented into domain-specific weights and domain-agnostic weights. Only domain-specific weights are updated during training for a particular domain, while domain-agnostic weights remain frozen to preserve knowledge from previously trained domains. This segmentation allows the model to adapt to new domains without forgetting previously learned information.
Solution Approach 2:
The invention extracts and isolates only the necessary domain-specific weight components that need to be updated for each domain, rather than updating the entire weight matrix. This extraction approach minimizes the scope of weight updates to only what is necessary for domain adaptation, preventing interference with previously trained data.
2Loss of information
If build from scratch scheme using entire corpus including new data is used, then the forgetting issue is solved, but computational resources and training time are consumed excessively
Solution Approach 1:
Instead of performing complete retraining from scratch with the entire corpus, the invention applies partial action by updating only the domain-specific weight portions. This partial update approach achieves the necessary adaptation to new domains while avoiding the excessive computational cost and time consumption of full retraining.
Solution Approach 2:
The training process applies local quality by focusing computational resources only on the specific weight components relevant to the current domain being trained, rather than uniformly processing the entire model. This localized approach to weight updates significantly reduces computational resources and training time while maintaining effectiveness.
3Adaptability or versatility
If the number of domains increases, then the number of language models corresponding to a domain needs to be increased, but resource and time consumption increases excessively
Solution Approach 1:
The language model is designed with universal domain-agnostic weight components that can serve multiple domains, combined with domain-specific weight components that can be selectively updated. This multi-functional architecture allows a single language model to handle multiple domains without requiring separate models for each domain, thereby reducing overall resource consumption.
Solution Approach 2:
The model weights are segmented into shared domain-agnostic components and domain-specific components. This segmentation enables efficient resource utilization by allowing the shared components to serve multiple domains while only the domain-specific components need to be updated when adding new domains, rather than duplicating entire models.
Data Source
AI summary
Disclosed is an electronic device including processor and memory operatively connected to the processor and storing language model. The electronic device may enter data into the language model, generate an embedding vector in the input embedding layer, add position information to the embedding vector in the positional encoding layer, branch the embedding vector based on domain information, normalize the branched embedding vectors, enter the normalized embedding vectors into the multi-head attention layer, enter output data of the multi-head attention layer into the first layer, normalize pieces of output data of the first layer, enter the normalized pieces of output data of the first layer into the feed-forward layer, enter output data of the feed-forward layer into the second layer and normalize pieces of output data of the second layer, and enter the normalized pieces of output data of the second layer into the linearization layer and the softmax layer to obtain result data. In addition, various embodiments as understood from the specification may be also possible.