Single-Stream Vision-Language Pretraining With Multi-Level Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vision-language pretraining techniques suffer from weak alignment between visual and text modalities due to simple architectures and loss functions, relying heavily on large datasets and compute infrastructure, which fail to effectively align image and text features at multiple levels of semantic abstraction.
Innovation Solution
A single stream model employing multi-level alignment through symmetric cross-modality reconstruction, concept prediction, and self-supervised keyword pseudo labeling, using a deep stack of transformer encoder layers and a multi-task loss to align visual and text representations at global, fine-grained, and conceptual semantic levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing simple architecture and loss function are used for vision-language pretraining, then the model can be trained with large datasets and compute infrastructure, but the alignment between visual and text modalities remains weak
Solution Approach 1:
The patent segments the alignment process into multiple levels: global instance level alignment using contrastive loss, fine-grained token level alignment using reconstruction loss, and conceptual semantic level alignment using concept prediction loss. This multi-level segmentation allows the model to achieve strong alignment without requiring excessively complex architecture by addressing different aspects of alignment separately at appropriate granularities.
Solution Approach 2:
The patent adds a new dimension to the alignment problem by introducing multi-level alignment across different semantic abstractions (global, fine-grained, conceptual) rather than relying solely on increasing model size or data quantity. This dimensional approach to alignment enables stronger modality coupling through diverse loss functions operating at different representational levels.
2Quantity of substance
If large datasets with hundreds of millions of image-text pairs are used, then more training data is available, but the compute infrastructure requirements increase significantly
Solution Approach 1:
The patent employs self-supervised learning mechanisms where the model learns from the structure and content of the data itself rather than requiring extensive external supervision. The symmetric cross-modality reconstruction and concept prediction tasks enable the model to generate its own training signals from the image-text pairs, reducing the need for additional annotated data and lowering compute requirements compared to fully supervised approaches.
Solution Approach 2:
The patent changes the optimization parameters by introducing multi-task loss functions that operate at different levels of semantic abstraction. This allows more efficient utilization of available data by extracting multiple learning signals from each image-text pair through different loss components (contrastive loss, reconstruction loss, concept prediction loss), thereby reducing the total number of training iterations and compute resources needed.
3Ease of manufacture
If simple architecture is used for vision-language pretraining, then the model is easier to train, but the alignment at multiple levels of semantic abstraction is insufficient
Solution Approach 1:
The patent makes the encoder architecture universal by using the same visual encoder and text encoder for multiple functions: global instance alignment, fine-grained token alignment, and conceptual concept prediction. This multi-functional design allows a single architecture to achieve alignment at multiple semantic levels without requiring separate specialized models, thereby maintaining ease of training while improving alignment precision through the multi-task objective.
4Productivity
If existing pretraining techniques are used, then the model can be trained with current resources, but effective alignment for downstream tasks like image captioning and visual question answering is not achieved
Solution Approach 1:
The patent performs preliminary alignment at multiple semantic levels during the pretraining phase before downstream task training. By establishing strong global, fine-grained, and conceptual alignment between visual and text modalities in advance, the model is better prepared for downstream tasks like image captioning and visual question answering, improving their performance without requiring complex task-specific architectures.
Data Source
AI summary
A method is provided for pretraining vision and language models that includes receiving image-text pairs, each including an image and a text describing the image. The method encodes an image into a set of feature vectors corresponding to input image patches and a CLS token which represents a global image feature. The method parses, by a text tokenizer, the text into a set of feature vectors as tokens for each word in the text. The method encodes the CLS token from the NN based visual encoder and the tokens from the text tokenizer into a set of features by a NN based text and multimodal encoder that shares weights for encoding both the CLS token and the tokens. The method accumulates the weights from multiple iterations as an exponential moving average of the weights during the pretraining until a predetermined error threshold is reduced to be under a threshold amount.