Grounded Image Context Queries with Fewer Large Model Calls

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing large generative models (LGMs) face inaccuracies and inefficiencies when generating text query responses for unique or recently posted images, often requiring excessive computational resources and providing unsatisfactory results.

Innovation Solution

An image query system that utilizes a comprehensive image prompt to obtain visual image grounding information from a visual-based large generative model, supplemented by reverse image search grounding information, to generate accurate text query responses efficiently, minimizing calls to the computationally expensive visual-based model.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the visual-based large generative model is called repeatedly to generate text query responses for unique or recently posted images, then the accuracy of the responses is improved, but the computational resources consumed increase excessively

Engineering Contradiction:

Improveaccuracy of text query responsesVSAvoidcomputational resources consumed

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs reverse image search and extracts grounding information in advance before generating text responses. This preliminary action retrieves relevant image data, captions, and metadata that can be used to answer queries about unique or recently posted images without requiring repeated calls to the visual-based large generative model, thus reducing computational resource consumption while maintaining response accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary layer that uses grounding information from reverse image search as a bridge between the input image and the text generation process. This intermediary grounding information serves as a cache or reference that enables the system to generate accurate text responses without repeatedly invoking the computationally expensive visual-based large generative model, effectively mediating between accuracy requirements and computational constraints.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If the visual-based large generative model is called multiple times for different queries about the same image, then the comprehensiveness of information is improved, but the computational overhead increases

Engineering Contradiction:

Improvecompleteness of image informationVSAvoidcomputational overhead

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The system performs reverse image search and extracts comprehensive grounding information in advance, including image captions, metadata, and related content. This preliminary extraction of complete image information allows the system to answer multiple different queries about the same image using the pre-retrieved grounding information, eliminating the need for repeated calls to the visual-based large generative model and significantly reducing computational overhead while maintaining information comprehensiveness.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The grounding information retrieved through reverse image search serves multiple functions simultaneously: it provides context for generating text responses, enables answer verification, and supports various types of queries about the same image. This multi-functional use of grounding information allows the system to handle diverse queries without repeatedly invoking the visual-based large generative model, thereby reducing computational overhead while maintaining comprehensive information coverage.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If grounding information is used to generate text query responses, then the accuracy for unique and recently posted images is improved, but the system complexity increases

Engineering Contradiction:

Improveaccuracy of responses for unique imagesVSAvoidsystem architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system introduces grounding information as an intermediary component that bridges the input image and the text generation process. This intermediary layer, populated by reverse image search results, provides structured context that enables accurate responses for unique and recently posted images. By organizing the complexity into a separate grounding information retrieval and storage layer, the system manages architectural complexity while improving response accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12405995B2Using large generative models with improved grounding to improve image context queries

Publication Date: 2025.09.02 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The disclosure describes utilizing an image query system to improve response accuracy and reduce computational steps resources in responding to natural language queries of input images. In various implementations, the image query system utilizes grounding information from one or more sources to determine accurate information for an input image. For example, the image query system uses a single comprehensive image prompt to obtain extensive visual image grounding information for the input image from a visual-based large generative model. Additionally, or in alternative implementations, the image query system obtains reverse image search grounding information for the input image. The image query system then cleverly utilizes the grounding information with a large generative language model to generate text query responses to image-based queries of the input image more accurately and efficiently.