Multi-Modal Text Embedding Training for Paraphrase-Robust Image Retrieval

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional multi-modal encoders struggle to handle linguistic variations in text queries, such as paraphrases, leading to inaccurate image retrieval results due to distantly spaced text embeddings for similar content phrased differently.

Innovation Solution

Train a multi-modal encoder to generate text embeddings based on the similarity between an image caption and its paraphrase, allowing for closer embeddings for semantically similar text prompts.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional multi-modal encoders are used to generate text embeddings, then the encoding process is simple and fast, but the text embeddings for paraphrases of the same content are distantly spaced, leading to inaccurate image retrieval

Engineering Contradiction:

Improvetext embedding similarity accuracyVSAvoidencoder training complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The multi-modal encoder is pre-trained using paraphrase-caption pairs from training data before actual image retrieval. This preliminary training step teaches the encoder to map semantically equivalent paraphrases to closely spaced embeddings, resolving the contradiction by preparing the system in advance to handle linguistic variations accurately

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The training process modifies the encoder's internal parameters (weights and biases) by optimizing for paraphrase-caption similarity. This parameter adjustment enables the encoder to generate closely spaced embeddings for paraphrases while maintaining efficient inference, thus improving measurement precision without permanently increasing operational complexity

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If conventional multi-modal encoders generate text embeddings, then the retrieval process is quick, but different users querying the same content with different phrasing obtain different image results

Engineering Contradiction:

Improvelinguistic variation handlingVSAvoidretrieval consistency

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The encoder is trained using feedback from paraphrase-caption similarity measurements. During training, the system evaluates how closely spaced the embeddings of paraphrase pairs are and adjusts parameters to minimize distance. This feedback mechanism enables the encoder to consistently handle linguistic variations, improving both adaptability and retrieval reliability

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The multi-modal encoder is designed to handle multiple text formulations (paraphrases) that convey the same meaning, making it universal in dealing with linguistic variations. This multi-functionality in text processing ensures that different user queries about the same content produce consistent retrieval results, resolving the contradiction between adaptability and reliability

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20260037572A1Text-based image retrieval

Publication Date: 2026.02.05 ADOBE INC

AI summary

A method, apparatus, non-transitory computer readable medium, and system for media processing include obtaining a text prompt describing content, generating, using a multi-modal encoder, a text embedding based on the text prompt, and obtaining an image depicting the content based on the text embedding. The multi-modal encoder is trained to encode image descriptions based on a similarity between a caption of a training image and a paraphrase of the caption.