Multimodal Search Model Integrating Text Refinements for Image Retrieval

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional visual search technologies cannot incorporate textual query refinements, leading to inefficient image retrieval and user frustration due to incorrect determination of user intent, resulting in unnecessary resource usage and inability to retrieve visually similar images effectively.

Innovation Solution

A machine-learned multimodal search refinement method that processes query image embeddings and textual query refinements using a machine-learned query refinement model to generate refined image embeddings, allowing for efficient retrieval of images by modifying model parameters based on a loss function evaluating the distance between refined embeddings and ground truth images.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional visual search services process images using machine learning techniques to identify visually similar images, then image retrieval functionality is provided, but the system cannot incorporate textual query refinements leading to incorrect determination of user intent

Engineering Contradiction:

Improveability to incorporate textual query refinementsVSAvoidaccuracy in determining user intent

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent combines image embeddings and textual query refinements into a unified multimodal search model. The model processes both visual features from the query image and textual features from user refinements simultaneously, merging these different modalities to produce a comprehensive search result that accurately reflects user intent.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The search system is enhanced to perform multiple functions: it processes image embeddings for visual similarity search, incorporates textual query refinements for semantic understanding, and integrates both modalities to provide a unified search interface that handles diverse user queries effectively.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If conventional models are trained only to process image data, then image processing capability is maintained, but the system lacks the ability to process textual query refinements

Engineering Contradiction:

Improvemultimodal processing capabilityVSAvoidmodel architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the processing of different modalities into separate components: an image embedding processor that handles visual data and a textual query processor that handles text refinements. These segmented components are then integrated through a loss function that combines their outputs, allowing the system to process multiple modalities while maintaining clear functional boundaries.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a loss function as an intermediary mechanism that bridges the image processing and text processing components. This loss function evaluates the combined representation from both modalities and guides the training process, serving as a mediator that enables multimodal processing without requiring complete architectural integration.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If visual search services provide initial result images without textual refinement capability, then simple image retrieval is achieved, but user experience deteriorates due to inability to refine queries

Engineering Contradiction:

Improvequery refinement capabilityVSAvoidtime for multiple search iterations

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The system performs preliminary processing of both image and text inputs through the multimodal model before generating search results. By pre-integrating the textual query refinement with the image embedding in the loss function, the system prepares a comprehensive search representation in advance, eliminating the need for multiple iterative searches and reducing user wait time.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20240370487A1Machine-Learned Models for Multimodal Searching and Retrieval of Images

Publication Date: 2024.11.07 GOOGLE LLC

AI summary

Systems and methods of the present disclosure are directed to computer-implemented method for machine-learned multimodal search refinement. The method includes obtaining a query image embedding for a query image and a textual query refinement associated with the query image. The method includes processing the query image embedding and the textual query refinement with a machine-learned query refinement model to obtain a refined query image embedding that incorporates the textual query refinement. The method includes evaluating a loss function that evaluates a distance between the refined query image embedding and an embedding for a ground truth image within an image embedding space. The method includes modifying value(s) of parameter(s) of the machine-learned query refinement model based on the loss function.