Multimodal Submodel Local Data Locating for Fine-Grained Correspondence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multimodal deep learning pre-training processes focus on global-dimensional feature contrast, neglecting finer-grained correspondences between modal data, which limits the performance of pre-trained models in downstream tasks.
Innovation Solution
The proposed method involves a processing model with a multimodal submodel that includes a pre-training task of locating local data matching between different modalities, establishing finer-grained correspondence relationships, thereby improving model performance on tasks like video-text localization, retrieval, and generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pre-training focuses on global-dimensional feature contrast, then the model can process multimodal data, but the performance on downstream tasks is limited due to ignoring finer-grained correspondences
Solution Approach 1:
The patent segments the feature space by introducing local feature extractors that operate on specific regions or segments of multimodal data (e.g., local image patches, text spans, or audio segments). This segmentation enables the model to capture fine-grained correspondences between different modalities at the local level, rather than only global-level alignments.
Solution Approach 2:
The patent applies local quality by making different parts of the model handle different levels of abstraction: global feature extractors handle overall semantic alignment while local feature extractors handle detailed regional correspondences. This allows the system to maintain both global context awareness and local detail sensitivity, improving downstream task performance.
2Reliability
If the model learns fine-grained local correspondences, then downstream task performance improves, but the model complexity increases
Solution Approach 1:
The patent merges multiple feature extraction pathways (global and local) into a unified multimodal processing framework. By combining these extractors and their corresponding contrastive learning objectives into a single integrated model, the system achieves fine-grained correspondence learning without requiring separate complex systems for each level of analysis.
Solution Approach 2:
The patent designs universal feature extractors that can operate at multiple levels of granularity. The same architectural components (encoders, projectors, contrastive loss functions) are used for both global and local feature extraction, allowing the model to handle different levels of correspondence learning with a single multi-functional framework rather than requiring separate specialized systems.
Data Source
AI summary
The embodiments of the disclosure provides a processing method, apparatus, electronic device and non-transitory computer-readable storage medium for multimodal data, wherein the method includes: obtaining data to be processed of an original modality; determining result data of a target modality corresponding to the data to be processed by processing the data to be processed with a target processing model; wherein the target processing model comprises a multimodal submodel, and the pre-training task of the multimodal submodel includes a task of locating local data that matches second modal data from first modal data; wherein when the first modal data belongs to the original modality, the second modal data belongs to the target modality; when the first modal data belongs to the target modality, the second modal data belongs to the original modality.