Multi-Modal Model Training with Back Translation for Data Shift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenges of small training sample quantities and large distribution deviations in multi-modal model training lead to convergence difficulties and over-fitting, hindering the robustness of the model.
Innovation Solution
A multi-modal model training method that involves obtaining a training sample set with pairs of visual and textual data, performing back translation on the textual data using a sequence of translation components, and randomly selecting a back-translated text to augment the training samples, thereby increasing sample quantity and reducing distribution deviation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If back translation is performed on target original text through multiple translation components, then the quantity of training sample pairs increases and model convergence speed improves, but the device complexity and computational resources required increase
Solution Approach 1:
The translation module is divided into multiple independent translation components connected in sequence, where each component performs a specific translation task. This segmentation allows the system to process text through multiple transformation stages (original text → intermediate translation → back-translation), effectively generating diverse training samples while maintaining modular architecture that manages complexity.
Solution Approach 2:
The system performs back-translation as a preliminary data preparation step before actual model training. By pre-processing the training text through translation components to generate augmented samples, the system prepares enhanced training data in advance, which accelerates subsequent model convergence without adding complexity during the training execution phase.
2Reliability
If back translation is performed to augment training samples, then the quantity of training data increases and over-fitting is reduced, but the training time and computational resources increase
Solution Approach 1:
Back-translation is performed as a preliminary data augmentation step before model training begins. By pre-generating augmented training samples through translation components, the system prepares a more robust and diverse training dataset in advance, which reduces over-fitting during training while the computational cost is incurred during data preparation rather than during the actual training process.
Solution Approach 2:
The system creates copies of original training samples through back-translation processes. Instead of generating entirely new training data, the system translates original text through intermediate languages and reconstructs it, creating variant copies that preserve the semantic meaning while providing diversity. This copying approach efficiently increases sample quantity without requiring proportional increases in computational resources during training.
Data Source
AI summary
Provided are a multi-modal model training method, apparatus and device, and a storage medium. The method includes the following steps: obtaining a training sample set, and training a multi-modal model for a plurality of rounds by successively using each of training sample pair in the training sample set: during use of any one of the training sample pairs for training, obtaining an image feature of a target visual sample firstly, and then determining whether back translation needs to be performed on a target original text; when back translation needs to be performed on the target original text, performing corresponding back translation to obtain a target back-translated text, and obtaining a text feature of the target back-translated text; and training the multi-modal model based on the image feature and the text feature.