Embedding Model Training via Local-Global View Contrast
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for embedding multi-modal data in deep learning require large amounts of paired datasets, leading to high costs for training set acquisition, preprocessing, and quality verification, and are performance-dependent.
Innovation Solution
A method involving the generation of view samples corresponding to local and global views of reference samples, where the first embedding model is updated based on the difference between outputs from itself and a second embedding model, reducing the need for extensive training sets and improving performance through data augmentation techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of paired datasets are used for training multi-modal deep learning models, then model performance is improved, but cost for training set acquisition, preprocessing, and quality verification increases significantly
Solution Approach 1:
The patent segments the training process into two stages: pre-training on single-modal data and fine-tuning on multi-modal data. This allows the model to first learn robust representations from abundant single-modal data, then adapt to multi-modal relationships with smaller paired datasets, thereby reducing the quantity of training data needed while maintaining performance
Solution Approach 2:
The patent performs preliminary pre-training on single-modal data before fine-tuning on multi-modal data. This preliminary action enables the model to acquire fundamental embedding capabilities from easily obtainable single-modal data, which then serves as a foundation for more complex multi-modal tasks with smaller datasets
2Measurement precision
If large amounts of training sets are used for multi-modal embedding, then embedding accuracy is improved, but cost for securing and preprocessing training sets increases
Solution Approach 1:
The patent segments data acquisition and processing into single-modal data collection (easy, abundant) and multi-modal paired data collection (difficult, limited). By training on single-modal data first and then fine-tuning on multi-modal pairs, the system achieves high embedding accuracy without requiring extensive preprocessed multi-modal training sets
Solution Approach 2:
The patent uses single-modal data as a proxy or copy to pre-train the model before using actual multi-modal paired data for fine-tuning. This copying approach allows the model to learn from readily available single-modal data representations, which then transfer to multi-modal tasks with smaller, more expensive to acquire paired datasets
Data Source
AI summary
Provided are a method for generating a summary and system therefor. The method according to some embodiments may include generating a first view sample corresponding to a local view of a reference sample. generating a second view sample corresponding to a view greater than the local view from the reference sample; generating a first output value by inputting the first view sample to a first embedding model. generating a second output value by inputting the second view sample to a second embedding model. and updating parameters of the first embedding model based on a difference between the first output value and the second output value.