Multi-modal Neural Network Image Ranking via Visual and Textual Fusion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional image searching technologies fail to accurately rank images against queries due to overlooking specific image regions and neglecting rich metadata such as captions, tags, and keywords, leading to difficulties in recognizing visually distorted or partially relevant images.

Innovation Solution

A multi-modal neural network is employed to evaluate similarities between queries and images, combining visual and textual features by determining importance weightings for each modality, allowing for more accurate rankings by emphasizing textual features when visual features are unreliable and vice versa.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If conventional neural networks classify overall images, then processing is simplified, but specific relevant regions and metadata are overlooked

Engineering Contradiction:

Improveprocessing complexityVSAvoidmetadata and region information loss

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent divides the image into multiple regions and processes each region separately through region-specific neural networks, rather than classifying the overall image. This segmentation allows the system to capture specific relevant regions while maintaining manageable processing complexity through modular architecture.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent combines multiple modalities including visual features from segmented regions, textual features from metadata (captions, tags, keywords), and spatial relationships into a unified ranking framework. This merging integrates previously separate information sources to prevent information loss while maintaining systematic processing.

Inventive Principle:

Principle #5Merging (Combining)

2Device complexity

If conventional approaches use single modality evaluation, then system complexity is reduced, but accuracy in recognizing distorted or partial images deteriorates

Engineering Contradiction:

Improvesystem complexityVSAvoidimage relevance recognition accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent dynamically adjusts the importance weighting of different modalities (visual, textual, spatial) based on the specific query and image characteristics. This parameter change allows the system to adapt to distorted or partial images by emphasizing more reliable modalities, improving recognition accuracy without requiring a completely complex system redesign.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent creates a composite evaluation framework that combines multiple modalities (visual features, textual metadata, spatial relationships) with different importance weightings. This composite approach resembles composite materials where different components work together to achieve properties that individual components cannot provide alone, improving accuracy for distorted or partial images.

Inventive Principle:

Principle #40Composite materials

3Ease of operation

If conventional networks rely solely on visual features, then processing is straightforward, but reliability decreases when visual features are distorted or unreliable

Engineering Contradiction:

Improveprocessing simplicityVSAvoidranking reliability under visual distortion

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent introduces textual metadata (captions, tags, keywords) as an intermediary modality that can compensate for unreliable visual features. When visual features are distorted, the textual information serves as a mediator to maintain ranking reliability, bridging the gap between query and image evaluation without complicating the overall processing flow.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent implements dynamic importance weighting that adjusts the reliance on visual versus textual modalities based on the specific evaluation context. This dynamic adjustment allows the system to maintain processing simplicity while improving reliability by automatically emphasizing more trustworthy modalities when visual features are distorted or unreliable.

Inventive Principle:

Principle #15Dynamics

4Productivity

If conventional approaches overlook metadata, then processing speed is maintained, but ranking accuracy deteriorates

Engineering Contradiction:

Improveprocessing speedVSAvoidranking accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent performs preliminary extraction and processing of metadata (captions, tags, keywords) alongside visual features, preparing all modalities in advance for the ranking evaluation. This preliminary action ensures that metadata is readily available when needed, maintaining processing speed while improving ranking accuracy through comprehensive information utilization.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10642887B2Multi-modal image ranking using neural networks

Publication Date: 2020.05.05 ADOBE INC

AI summary

Methods and systems are provided for ranking images against queries. A visual modality ranking of visual features of a digital image against a query is generated. A language modality ranking of text features of text associated with the digital image against the query is also generated. A multi-modal neural network determines importance weightings of the language modality ranking and the visual modality ranking against the query. The visual modality ranking and the language modality ranking are combined into a multi-modal ranking of the digital image against the query based on the importance weightings. The digital image is provided as a search result of the query based on the multi-modal ranking.