Cross-Modal Search Embedding Bins and Extended Relevance Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cross-modal search systems face challenges in effectively training embedding functions to accurately match search queries across different modalities, such as text and images, leading to suboptimal search results due to limited relevance matrices that do not adequately capture similarities between text captions and associated items.
Innovation Solution
The system extends relevance matrices by selectively adding additional relevance values based on similarities between text captions, trains embedding functions using triplet losses, listwise losses, or quantized metrics like mean average precision (mAP) and normalized discounted cumulative gain (NDCG), and splits the embedding space into bins to improve ranking measures and loss determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If relevance matrices are extended by adding additional relevance values based on text caption similarities, then search result accuracy and relevance are improved, but system complexity and computational requirements increase
Solution Approach 1:
The patent segments the relevance representation into multiple matrices: the original relevance matrix Rvt (image-to-text) and an extended relevance matrix Rvt' that incorporates text caption similarities. By dividing the relevance modeling into distinct components, the system can maintain the original simple structure while adding complexity only where needed for improved accuracy.
Solution Approach 2:
The patent extends the relevance matrix by adding another dimension of information - text caption similarities - to the existing image-text relevance relationships. This creates an extended matrix Rvt' that combines direct image-text relevance with indirect relevance propagated through similar text captions, thereby improving search accuracy without completely redesigning the system.
2Productivity
If embedding space is split into bins for improved ranking, then retrieval performance is improved, but computational overhead increases
Solution Approach 1:
The patent segments the continuous embedding space into discrete bins based on distance thresholds from query embeddings. This segmentation allows the system to process and rank results in structured intervals, improving retrieval performance by enabling more nuanced ranking while managing computational complexity through the discretization of the embedding space.
3Measurement precision
If multiple loss functions are used for training embedding functions, then training accuracy is improved, but training complexity increases
Solution Approach 1:
The patent merges multiple loss functions - triplet loss, listwise loss, and quantized loss - into a unified training framework. By combining these different loss functions that operate at different levels (instance-level, list-level, and metric-level), the system achieves comprehensive training accuracy improvement while managing complexity through integrated optimization.
Data Source
AI summary
A system includes: a training dataset including first objects of a first modality and second objects of a second modality different than the first modality, where the second objects include text that is descriptive of the first objects; a first matrix including first relevance values indicative of relevance between the first objects and the second objects, respectively; a second matrix including second relevance values indicative of relevance between the second objects and the first objects, respectively; and a training module configured to: assign ones of the second objects to bins based on distances between the ones of the objects and a query; determine a ranking measure based on a number of the ones of the second objects assigned to the bins; determine losses based on the ranking measure and the first and second matrices; determine a final loss based on the losses; train embedding functions based on the final loss.