Language Model Training with Dual Word Vector Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing self-supervised pre-training process of language models in natural language processing, based on character granularity, faces challenges in learning information with greater semantic granularity, leading to an information leakage risk and compromised prediction performance.
Innovation Solution
A method and apparatus for training a language model using various word vectors, where a first and second pre-trained word vector parameter matrices are combined to determine a word vector for a word mask, allowing the model to learn multi-source high-quality word meaning information and avoid information leakage risks associated with character-based training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the language model is trained based on character granularity to prevent influence from word segmenter performance, then the training effect is protected, but the model's ability to learn semantic information at word level is compromised and information leakage risk increases
Solution Approach 1:
The patent merges character-level training and word-level training by constructing training samples that contain both character sequences and corresponding word annotations. The model simultaneously processes character inputs while learning word-level semantic representations through the word vector module, thereby combining the stability of character-level training with the semantic richness of word-level information without causing information leakage.
2Ease of manufacture
If the language model is trained based on character granularity, then the training process is simplified, but the capacity to learn word meaning information is reduced
Solution Approach 1:
The patent segments the training process into two distinct modules: a character-level language model module that maintains training simplicity, and a word vector module that enhances word meaning learning capacity. The word vector module separately processes word-level semantic information and provides supplementary training signals, allowing the overall system to maintain simplicity while gaining enhanced word understanding capabilities.
3Reliability
If character-based training is used to avoid word segmenter performance influence, then training robustness is improved, but prediction performance is compromised
Solution Approach 1:
The patent introduces a word vector module as an intermediary component that bridges character-level processing and word-level semantics. This module acts as a mediator by transforming character sequences into word-level representations and providing supplementary training signals that enhance prediction performance while maintaining the robustness benefits of character-based training through the primary language model module.
Data Source
AI summary
A method for training a language model based on various word vectors, a device and a medium, which relate to the field of natural language processing technologies in artificial intelligence, are disclosed. An implementation includes inputting a first sample text language material including a first word mask into the language model, and outputting a context vector of the first word mask via the language model; acquiring a first probability distribution matrix of the first word mask based on the context vector of the first word mask and a first word vector parameter matrix, and a second probability distribution matrix of the first word mask based on the context vector of the first word mask and a second word vector parameter matrix; and training the language model based on a word vector corresponding to the first word mask.