Visual Question Answering Image Retrieval for Condition-Based Scene Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image retrieval technologies using keywords or similarity-based methods fail to effectively retrieve images or scenes that do not meet specific conditions, such as wearing a helmet, from moving images captured on-site or in sharing sites.
Innovation Solution
An information processing system that converts user queries into structured questions, utilizes a visual question answering model to estimate answers for each image or frame, and groups images based on these answers to facilitate retrieval of desired scenes or conditions, including the use of visual question answering (VQA) models and grouping units to classify and display representative images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional image retrieval technologies (keyword-based or similarity-based) are used, then the system is simple to operate, but it cannot retrieve images or scenes that do not meet specific conditions
Solution Approach 1:
The patent introduces a visual question answering model as an intermediary component between the user query and the image retrieval system. This model converts natural language queries into structured questions and generates answer estimations that guide the retrieval process, enabling the system to handle negative condition queries while maintaining operational simplicity for users
Solution Approach 2:
The patent segments the image retrieval process into distinct functional modules: a question conversion unit that transforms queries into structured questions, a visual question answering model that processes these questions, and a retrieval unit that executes the search. This segmentation allows each component to specialize in specific tasks, improving overall versatility while keeping the system architecture manageable
2Measurement precision
If visual question answering models are used to estimate answers for each image, then the retrieval accuracy for specific conditions is improved, but the processing time and computational resources increase
Solution Approach 1:
The patent applies partial action by having the visual question answering model process only a subset of images or use sampled data from the image database rather than exhaustively analyzing every image. This approach achieves sufficient retrieval accuracy for specific conditions while significantly reducing the overall processing time and computational resource requirements
Solution Approach 2:
The system performs preliminary processing by pre-computing features or answer estimations for images and storing them in advance. When a retrieval query is received, the system can quickly reference these pre-computed results rather than performing full visual question answering analysis in real-time, thereby improving retrieval accuracy while minimizing processing time
3Ease of operation
If images are classified into multiple groups based on answer estimations, then the retrieval of desired scenes is facilitated, but the device complexity increases
Solution Approach 1:
The patent extracts and displays only the most relevant retrieval results or representative images from each group to the user, rather than presenting all classified images. This extraction approach facilitates easy operation by showing users precisely what they need while keeping the classification mechanism operational in the background without overwhelming the user interface
Data Source
AI summary
According to one embodiment, an information processing method includes acquiring a first question corresponding to an input query, estimating an answer to the first question for each of pieces of content to generate a first answer estimation result including the answer to the first question for each of the pieces of content, and classifying the pieces of content into first groups by using the first answer estimation result and generating a first retrieval result indicating content belonging to each of the first groups.