Multi-modal Pre-training Model Fine-grained Semantic Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multi-modal pre-training models fail to effectively distinguish between common and fine-grained semantic words in text, leading to poor training outcomes due to the lack of differentiation during the training process.
Innovation Solution
A method that determines and masks fine-grained semantic words in image-text pairs for training, allowing the model to learn these crucial words by replacing them with special characters or random tokens, and then trains the model using this modified data for improved cross-modal alignment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all words in text are treated equally during training, then the training process is simple, but the model cannot distinguish fine-grained semantic words leading to poor training effect
Solution Approach 1:
The patent segments text into common words and fine-grained semantic words, applying different processing strategies to each type. This segmentation enables the model to focus on distinguishing important semantic words while maintaining training efficiency through selective masking of only the fine-grained semantic words.
Solution Approach 2:
The patent applies local quality by treating different word types differently during training. Specifically, fine-grained semantic words are masked for training while common words are preserved, allowing the model to develop specialized attention to semantically important words without uniformly complicating the entire training process.
2Reliability
If fine-grained semantic words are masked during training, then the model learns better semantic representations, but the training data preparation becomes more complex
Solution Approach 1:
The patent applies preliminary action by pre-processing text to identify and mark fine-grained semantic words before the training process begins. This preliminary identification step, potentially using scene graphs or semantic parsing, prepares the data in advance so that during training, only the pre-identified fine-grained words need to be masked, streamlining the overall process despite the added initial step.
3Manufacturing precision
If common words and fine-grained semantic words are not distinguished, then the training process is straightforward, but the model achieves poor alignment between image and text representations
Solution Approach 1:
The patent segments the text corpus into common words and fine-grained semantic words, enabling differentiated training approaches. This segmentation directly improves cross-modal alignment by ensuring the model focuses learning on semantically discriminative words that are crucial for precise image-text correspondence, while maintaining straightforward processing for common words.
Data Source
AI summary
A multi-modal pre-training model acquisition method, an electronic device and a storage medium, which relate to the fields of deep learning and natural language processing, are disclosed. The method may include: determining, for each image-text pair as training data, to-be-processed fine-grained semantic word in the text; masking the to-be-processed fine-grained semantic words; and training the multi-modal pre-training model using the training data with the fine-grained semantic words masked.