CROMDA Vision-Language Distribution Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vision-language pre-training models fail to exploit the global structure of datasets, leading to ineffective uni-modal representations where similar images and texts do not share similar representations, resulting in less effective alignment between cross-modal and uni-modal representations.
Innovation Solution
The CROss-Modal Distribution Alignment (CROMDA) model aligns global cross-modal representations by computing uni-modal similarities and generating a softmax-normalized distribution, leveraging the global structure of the dataset to ensure images with similar text representations are similar and vice versa, using feature queues and momentum feature representations to align distributions across modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing VLP methods pre-train image and text representations on very large-scale datasets, then the model can learn basic vision-language alignment, but the uni-modal representation becomes less effective where similar images are not enforced to have similar representations
Solution Approach 1:
The patent segments the representation learning into two distinct components: cross-modal alignment (matching image-text pairs) and uni-modal consistency (ensuring similar images/texts have similar representations). This is achieved by separately computing cross-modal contrastive loss and uni-modal distribution alignment loss, allowing each objective to be optimized independently while contributing to overall performance
Solution Approach 2:
The patent introduces a new dimension of optimization by computing distribution alignment across the entire feature queue, not just pairwise comparisons. By using softmax-normalized distributions and KL divergence, the method adds a global structural constraint that operates in the probability distribution space, ensuring consistent representations across the entire dataset
2Reliability
If VLP models align features for each image-text pair, then cross-modal matching is improved, but the global structural information in the training data is not utilized
Solution Approach 1:
The patent implements feedback by computing distribution alignment between the current batch features and the accumulated feature queue. The KL divergence calculation provides a feedback signal that guides the model to maintain consistent representations with previously seen similar samples, creating a continuous learning loop that preserves global structural information
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing feature queues from the large-scale dataset before fine-tuning. These pre-computed distributions serve as a reference framework that guides the alignment process, allowing the model to leverage global structure information without re-processing the entire dataset during training
3Ease of manufacture
If similar images and texts are not enforced to have similar representations, then model training is simpler, but retrieval effectiveness decreases
Solution Approach 1:
The patent changes the parameter space by working with softmax-normalized probability distributions rather than raw feature vectors. By computing KL divergence between distributions, the method creates a smooth, differentiable loss function that naturally enforces similarity constraints without requiring complex optimization algorithms, maintaining training simplicity while improving retrieval effectiveness
Data Source
AI summary
Embodiments described herein a CROss-Modal Distribution Alignment (CROMDA) model for vision-language pretraining, which can be used for retrieval downstream tasks. In the CROMDA mode, global cross-modal representations are aligned on each unimodality. Specifically, a uni-modal global similarity between an image/text and the image/text feature queue are computed. A softmax-normalized distribution is then generated based on the computed similarity. The distribution thus takes advantage of property of the global structure of the queue. CROMDA then aligns the two distributions and learns a modal invariant global representation. In this way, CROMDA is able to obtain invariant property in each modality, where images with similar text representations should be similar and vice versa.