Multi-Modal Image Search via Composite Visual Query Construction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile devices lack an effective platform for visual search without capturing a photograph, as text or voice input struggles to express complex visual intent, and existing search engines have limited ability to handle long queries or semantic meaning.
Innovation Solution
A multi-modal image search facility on mobile devices that uses touch input to extract lexical entities from spoken queries, matches them to image tags, and constructs a composite visual query by selecting and manipulating candidate images, allowing for context-aware image search.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If text or voice input is used for visual search on mobile devices, then the search can be performed without capturing a photograph, but the ability to express complex visual intent is insufficient
Solution Approach 1:
The system segments the visual search query into multiple lexical entities from the text/voice input, then retrieves and assembles multiple candidate images for each entity. This segmentation allows complex visual intent to be broken down into manageable components that can be independently searched and combined.
Solution Approach 2:
The system transitions from one-dimensional text/voice input to a two-dimensional composite image construction process. Users can manipulate candidate images in spatial dimensions (position, size, orientation) on a canvas, adding a visual-spatial dimension to the search process that better expresses complex visual intent.
2Productivity
If existing search engines handle long queries, then more complex search intents can be captured, but the machine learning of semantic meaning is insufficient
Solution Approach 1:
The system divides long queries into discrete lexical entities and processes each entity independently through image tag matching. This segmentation approach avoids the semantic understanding limitations of traditional machine learning by using structured entity extraction and independent image retrieval for each entity.
Solution Approach 2:
The system introduces an intermediary process between text input and image search results: lexical entity extraction and candidate image assembly. This intermediary layer translates natural language queries into structured visual representations without requiring advanced semantic understanding, bridging the gap between text and image domains.
3Ease of manufacture
If desktop search interfaces are adapted for mobile devices, then existing search functionalities can be maintained, but user interaction suitability is poor
Solution Approach 1:
The system implements a dynamic, multi-stage search process that adapts to mobile interaction constraints. Instead of a static desktop-like interface, the system dynamically progresses from text/voice input to entity extraction, candidate image presentation, and interactive canvas manipulation, allowing users to engage at their comfort level.
Solution Approach 2:
The search process is segmented into distinct mobile-friendly stages: input (text/voice), processing (entity extraction), intermediate display (candidate images), and final assembly (interactive canvas). This segmentation allows each stage to be optimized for mobile interaction without requiring full desktop functionality.
4Reliability
If capture-to-search is used, then photograph-based visual search is enabled, but the requirement of capturing a photograph limits search scenarios
Solution Approach 1:
Instead of starting with a photograph and searching for matches (capture-to-search), the system inverts the process by starting with text/voice description, extracting lexical entities, and constructing composite images from candidate matches. This inversion enables search scenarios where no photograph is available or appropriate.
Solution Approach 2:
The system creates composite images by copying and assembling candidate image fragments that match lexical entities from the query. These composite images serve as visual representations of the text-based query, enabling visual search without requiring an original photograph as input.
Data Source
AI summary
A facility for visual search on a mobile device takes advantage of multi-modal and multi-touch input on the mobile device. By extracting lexical entities from a spoken search query and matching the lexical entities to image tags, the facility provides candidate images for each entity. Selected ones of the candidate images are used to construct a composite visual query image on a query canvas. The relative size and position of the selected candidate images in the composite visual query image, which need not be an existing image, contribute to a definition of a context of the composite visual query image being submitted for context-aware visual search.