Semantic Chunk Weight Masking for Low-Data PLM Domain Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for domain-adapting pre-trained language models require large amounts of domain-specific data and struggle to reflect the relative importance and position-dependent characteristics of semantic chunks, leading to suboptimal performance, especially in low-resource domains.
Innovation Solution
A method using a domain semantic chunk graph and PageRank algorithm to select domain semantic chunks, combined with semantic chunk dynamic weight masking, which reflects the output value of the pre-trained language model, ensuring improved domain performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If continuous pre-training is used to achieve domain performance, then domain-specific accuracy can be improved, but a large amount of domain data is required which increases data requirements
Solution Approach 1:
The patent introduces an auxiliary model as an intermediary component that integrates domain N-grams and semantic chunk information. This auxiliary model acts as a mediator between the pre-trained language model and domain-specific data, allowing the system to achieve domain adaptation without requiring large amounts of domain training data. The auxiliary model processes domain-specific information and guides the main model's predictions.
Solution Approach 2:
The patent performs preliminary extraction and processing of domain N-grams and semantic chunks before the main training process. By pre-processing domain information into structured representations (N-gram frequencies, semantic chunk embeddings), the system prepares domain knowledge in advance, reducing the need for extensive domain-specific training data during continuous pre-training.
2Ease of operation
If T-DNA processes all semantic chunks in the same manner, then processing simplicity is maintained, but the relative importance between semantic chunks and position-dependent characteristics are not reflected
Solution Approach 1:
The patent applies local quality by treating different semantic chunks differently based on their characteristics. The auxiliary model computes importance scores for each semantic chunk based on domain N-gram frequencies and positional information. This allows the system to apply different processing weights to different semantic chunks, reflecting their relative importance and position-dependent characteristics rather than treating them uniformly.
Solution Approach 2:
The patent changes parameters dynamically based on semantic chunk properties. The importance score parameter is computed for each semantic chunk based on its domain-specific frequency and positional context. This parameter change allows the model to adaptively weight different semantic chunks during processing, improving the reflection of semantic importance while maintaining a relatively simple processing framework.
3Loss of information
If Pointwise Mutual Information method is used to identify semantic chunks, then collocation information can be identified, but the method needs to identify only amounts of information of two words and has difficulty utilizing external information when domain data is limited
Solution Approach 1:
The patent creates a universal auxiliary model framework that can process multiple types of domain information. The model is designed to integrate not only domain N-grams but also semantic chunk embeddings and external knowledge representations. This multi-functional auxiliary model can utilize various external information sources (different types of N-grams, semantic embeddings, knowledge graphs) making the system adaptable to different data scenarios and external information types.
Data Source
AI summary
A domain adaptation procedure, such as fine-tuning training, is required to utilize a large-capacity PLM for a specific domain. Attempts in existing research have been made to improve performance of a PLM through domain adaptor technology based on an N-gram in order to reduce errors on the basis of the results of domain text error analysis of the PLM. Proposed is a method of selecting a semantic chunk through a domain semantic chunk graph and PageRank based on the existing domain adaptor research, with an N-gram as the semantic chunk. Proposed is also a method of domain-adapting a large-capacity PLM using semantic chunk dynamic weight masking, which reflects an output value of a PLM rather than simply integrating embedding values of semantic chunks, in a semantic chunk domain adaptor technology.