Video-Text Feature Fusion for Fine-Grained Contrastive Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in multi-modal video understanding is the significant variation in content feature representation between different modalities, particularly between videos and text, necessitating a model that can effectively process and integrate diverse input data.
Innovation Solution
A data processing method that involves obtaining feature representations for videos and texts using dedicated encoders, fusing these representations, and performing contrastive learning to align and update the encoders, ensuring consistent processing granularity and incorporating both local and global information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If feature extraction is performed on text as a whole, then global text information is captured, but processing granularity is coarse and cannot match image-level processing
Solution Approach 1:
The patent segments the text into multiple text units (e.g., words, phrases, or sentences) and performs feature extraction on each unit individually. This segmentation allows the text processing granularity to match the image processing granularity (image block level), thereby improving processing precision. The segmented approach enables fine-grained alignment between text and image features while maintaining the ability to capture global context through the complete text sequence.
2Measurement precision
If only local text information is extracted, then processing precision is improved, but global text context is lost
Solution Approach 1:
The patent merges two types of feature representations: (1) local feature representations extracted from individual text units, and (2) global feature representation extracted from the complete text sequence. By combining these two sources of information, the system simultaneously achieves fine-grained processing precision and preserves global text context. The merged feature representation enables the model to understand both local details and overall meaning of the text.
3Adaptability or versatility
If different processing granularities are used for image and text branches, then each modality can be processed independently, but feature alignment and contrastive learning are compromised
Solution Approach 1:
The patent changes the processing parameter (granularity) of the text branch to match the image branch. Specifically, the text is segmented into units that correspond to image block granularity, and feature extraction is performed at this matched granularity level. This parameter adjustment enables direct alignment and contrastive learning between image and text features while maintaining the ability to process each modality independently through its own encoder.
Data Source
Figure 1A
Figure 1B~1C
Figure 1D~2
AI summary
A data processing method is disclosed, is applied to the field of video understanding in artificial intelligence, and includes: obtaining a video and text, where the text includes a plurality of text units; obtaining a first feature representation of the video based on the video by using an image encoder; obtaining, based on the text by using a text encoder, a second feature representation of each text unit and a third feature representation corresponding to the text, where the third feature representation is obtained by performing feature extraction by using the text encoder with the text as a whole; fusing the third feature representation and each second feature representation, to obtain a plurality of fourth feature representations; and performing contrastive learning between the first feature representation and the plurality of fourth feature representations, to update the image encoder and the text encoder. In this application, text information is fine-grained, so that processing precision of a network can be improved.