Cross-Modal Retrieval Training With Swapped Class Labels

The patent segments the training process into two independent modalities: image-based training and text-based training. Each modality is trained separately using its own feature extractor and loss function, avoiding the need to compute pairwise relationships across all data points. This segmentation reduces the training complexity from quadratic to linear while maintaining cross-modal retrieval capability through the shared embedding space.

The patent introduces an intermediary mechanism: class labels serve as mediators that bridge instances across different pairs. By predicting class labels for each instance independently and then using these labels to guide the training objective, the system can identify relevant instances across different pairs without making incorrect assumptions. This intermediary approach maintains training simplicity while improving relevance detection accuracy.

The patent applies self-service by enabling the model to generate its own training signals through self-supervised learning. Class labels are predicted automatically by the model itself during training, eliminating the need for manual annotation. These self-generated labels then serve as supervision signals to learn accurate similarity metrics, thus maintaining data preparation efficiency while improving similarity metric accuracy.