Multimodal Search Object Verification for Reliable LLM Responses

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large language models (LLMs) often generate hallucinations due to misinterpreting input data, leading to incorrect conclusions and undermining user trust, especially in critical applications.

Innovation Solution

A multimodal search system that receives image data and prompts, uses computer vision to identify objects, and requires user verification before generating responses using a large language model, incorporating reinforcement learning from human feedback (RLHF) to improve accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If a large language model generates responses directly from input data, then response generation speed is improved, but hallucination risk increases leading to reduced reliability

Engineering Contradiction:

Improveresponse generation speedVSAvoidresponse accuracy

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The system performs preliminary object identification using computer vision models before the LLM generates responses. This preliminary action verifies the actual content of input images, ensuring the LLM only processes confirmed information, thus preventing hallucinations while maintaining efficient response generation

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary verification layer between image input and LLM processing. This intermediary component validates image content through computer vision analysis, acting as a mediator that filters and confirms information before it reaches the LLM, thereby improving reliability without significantly slowing down the overall process

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If user verification is required for object identification, then response reliability is improved, but system complexity and processing time increase

Engineering Contradiction:

Improveresponse reliabilityVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system implements self-service through automated computer vision model verification that operates independently without requiring manual user intervention. The model automatically identifies and verifies objects in images, reducing system complexity while maintaining high reliability through automated validation processes

Inventive Principle:

Principle #25Self-service

3Reliability

If computer vision verification is performed before LLM processing, then hallucination is reduced, but computational overhead increases

Engineering Contradiction:

Improvehallucination reductionVSAvoidcomputational energy consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system applies partial verification by using computer vision models only for critical object identification tasks rather than comprehensive analysis of all image content. This selective approach reduces computational energy consumption while still effectively preventing hallucinations in key areas where accuracy is most crucial

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250272898A1User Verification of a Generative Response to a Multimodal Query

Publication Date: 2025.08.28 GOOGLE LLC

AI summary

A multimodal search system is described. The system can receive image data from a user device. Additionally, the system can receive a prompt associated with the image data. Moreover, the system can determine, using a computer vision model, a first object in the image data that is associated with the prompt. Furthermore, the system can receive, from the user device, a user indication on whether the image data includes the first object. Subsequently, in response to receiving the user indication, the system can generate a response using a large language model.