Fine-Grained Model Training with Coarse-Grained Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Masked language models, such as ERNIE and BERT-wwm, struggle to learn the semantic meaning of consecutive expressions due to independent predictions for each character, leading to low semantic closeness and inefficient calculation.
Innovation Solution
A method is introduced to process fine-grained samples into coarse-grained samples, annotate and mask them, and train a fine-grained model to learn content of a previous fine grain size and predict content of an adjacent coarse grain size, allowing for complete representation of coarse grain size through single prediction, thereby enhancing semantic meaning learning and model convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If independent predictions are performed for each character in masked language modeling, then the model can process fine-grained details, but the semantic meaning of consecutive expressions cannot be well learned and computational load increases
Solution Approach 1:
The patent segments the prediction task into two levels: fine-grained character-level predictions and coarse-grained expression-level predictions. By introducing coarse-grained samples that represent complete expressions or phrases, the model can perform segmented learning where it first learns character-level details and then learns to predict complete expressions as unified entities, thereby improving semantic closeness without proportionally increasing computational load
Solution Approach 2:
The patent adds a new dimension to the prediction task by introducing coarse-grained samples that operate at a higher level of abstraction. Instead of only predicting individual characters (fine-grained), the model now operates in two dimensions: character-level prediction and expression-level prediction. This dimensional expansion allows the model to capture semantic relationships at multiple granularities simultaneously
2Loss of information
If multiple fine-grained predictions are performed for consecutive expressions, then detailed character information is captured, but the overall semantic meaning learning is hindered and training convergence slows down
Solution Approach 1:
The patent applies preliminary action by pre-processing text data to generate both fine-grained character samples and coarse-grained expression samples before training. The coarse-grained samples are prepared in advance as complete expression units with their semantic annotations, allowing the model to learn expression-level patterns during training without requiring multiple iterative predictions, thus reducing training convergence time while preserving semantic meaning
Solution Approach 2:
The patent merges fine-grained character predictions and coarse-grained expression predictions into a unified training framework. By combining both types of samples in the training dataset and using a single model architecture that handles both granularities, the model learns to retain semantic meaning at the expression level while still capturing character-level details, avoiding the need for separate training processes
Data Source
AI summary
The present disclosure provides a method and apparatus for training a model. The method can include: acquiring at least one paragraph text, each paragraph text comprising a plurality of fine-grained samples; processing a fine-grained sample in the each paragraph text to obtain a coarse-grained sample; annotating the coarse-grained sample in the each paragraph text and obscuring one coarse-grained sample using a mask of one fine-grained sample to obtain a training sample set, wherein the training sample set comprises a plurality of annotated texts, and each annotated text comprises at least one of a fine-grained sample or an annotated coarse-grained sample; and training a fine-grained model using the training sample set to obtain a trained fine-grained model, the fine-grained model being used to learn content of a previous fine grain size and predict content of an adjacent coarse grain size.