Masked Language Model Training With Dynamic Mask Proportions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The current masked language model (MLM) training solution for pretrained language models (PLMs) results in undiversified training samples due to fixed mask proportions, leading to a bottleneck in natural language understanding capability and generalization.
Innovation Solution
Implement unfixed mask proportions at both text-level and word-level by using prior probability distribution models or neural network optimization to generate mask training samples, ensuring diverse training features and improved model convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If fixed mask proportion is used in MLM training, then training process is simple and easy to implement, but training samples become undiversified leading to bottleneck in natural language understanding capability
Solution Approach 1:
The patent applies dynamics by transitioning from fixed mask proportions to dynamic unfixed mask proportions. The mask proportion is no longer a static parameter but varies according to text characteristics, allowing the training process to adapt to different samples and enhance mode diversity while maintaining implementation feasibility.
Solution Approach 2:
The patent changes the parameter of mask proportion from a fixed value to an unfixed variable. By modifying this key parameter, the training samples gain diversity in mask proportions, which directly addresses the bottleneck in natural language understanding capability while keeping the overall training framework relatively simple.
2Stability of the object's composition
If fixed mask proportion is used in MLM training, then training sample generation is consistent and stable, but mode diversity is insufficient leading to poor generalization capability
Solution Approach 1:
The patent introduces dynamics into the mask proportion parameter, making it variable rather than fixed. This allows the training process to maintain stability in its overall structure while achieving diversity in specific samples, thereby improving generalization capability without completely abandoning consistency.
Solution Approach 2:
The patent applies local quality by allowing different mask proportions for different text samples based on their characteristics. Instead of applying a uniform mask proportion globally, the system tailors the mask proportion to each local sample, enhancing mode diversity and generalization while maintaining overall training stability.
3Adaptability or versatility
If unfixed mask proportions are used at text-level and word-level, then mode diversity and generalization capability are enhanced, but training process complexity increases
Solution Approach 1:
The patent changes the mask proportion from a fixed parameter to an unfixed parameter that can vary at both text-level and word-level. This parameter change enhances mode diversity and generalization capability. The complexity increase is managed by implementing this change within the existing MLM framework rather than redesigning the entire training process.
Solution Approach 2:
The patent segments the mask proportion parameter into two levels: text-level mask proportion and word-level mask proportion. This segmentation allows for controlled complexity by treating different aspects of masking separately, making the unfixed mask proportion system more manageable while still achieving enhanced diversity and generalization.
4Adaptability or versatility
If unfixed mask proportions are used in training, then natural language understanding capability is improved, but computational resources and training time may increase
Solution Approach 1:
The patent changes the mask proportion parameter to be unfixed, which improves natural language understanding capability. The trade-off in training time is accepted as a necessary cost for achieving better model performance and generalization, particularly in the context of large-scale pretraining where performance gains are prioritized.
Data Source
AI summary
Relating to the field of artificial intelligence, and specifically relating to the field of natural language processing, a data processing method includes and an apparatus performs: determining original text samples, where masking processing is not performed on the original text samples; and performing mask processing on the original text samples to obtain mask training samples, where the mask processing makes mask proportions of the mask training samples unfixed, and the mask training samples each are used to train a pretrained language model PLM. Training the PLM by using the mask training samples whose mask proportions are unfixed can enhance mode diversity of the training samples of the PLM. Therefore, features learned by the PLM are also diversified, a generalization capability of the PLM can be improved, and a natural language understanding capability of the PLM obtained through training can be improved.