Multimodal Query Input for Diversified Search Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search methods are inadequate in handling the diversification and increased volume of information, particularly in integrating non-text modalities such as images, sketches, and audio, leading to inefficiencies in information retrieval.
Innovation Solution
A multimodal search system that allows for the input and processing of queries comprising multiple types, including text, images, sketches, and audio, using a query input device that can convert and prioritize search results for diverse user needs, including accessibility for individuals with disabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If text-based search is used, then search simplicity is maintained, but ability to handle diversified information types is insufficient
Solution Approach 1:
The search system is designed to accept and process multiple types of query inputs including text, images, sketches, and audio signals through a single unified interface. The controller integrates these diverse input modalities and routes them to appropriate search algorithms, enabling the system to handle diversified information types without requiring separate search systems for each modality.
2Reliability
If multiple query types are accepted, then information retrieval effectiveness is improved, but processing complexity increases
Solution Approach 1:
The search system divides processing into distinct modules: text query processing, image query processing, sketch query processing, and audio query processing. Each module handles a specific query type using specialized algorithms, and the controller coordinates these modules. This segmentation allows the system to process multiple query types effectively while managing complexity through modular architecture.
Solution Approach 2:
The controller acts as an intermediary that receives diverse query inputs, determines the appropriate processing path for each query type, and coordinates the various search modules. It manages the complexity of handling multiple query types by centralizing the decision-making logic and providing a unified interface between different processing components.
3Adaptability or versatility
If image-based search is added, then search capability is enhanced, but system complexity increases
Solution Approach 1:
The system merges text-based search and image-based search into a single unified search platform. The controller integrates the text processing module and image processing module, allowing users to perform both types of searches through one system interface. This merging enhances search capability while avoiding the need for separate standalone systems.
4Adaptability or versatility
If accessibility features are added for disabled users, then user inclusivity is improved, but device complexity increases
Solution Approach 1:
The system provides multiple input modalities (text, image, sketch, audio) that serve both as enhanced search capabilities and as accessibility features. For example, audio input assists users with visual impairments, while sketch and image inputs assist users with writing difficulties. These multi-functional features improve user inclusivity without requiring separate accessibility systems.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
Provided are a method and an apparatus for searching for and acquiring information under a computing environment. The apparatus includes: at least one input device configured to receive a first query input of a first query type and a second query input of a second query type; and a controller configured to output a query input window including a first display item corresponding to the first query input and a second display item corresponding to the second query input, to automatically switch, in response to receiving the first query input, the apparatus from a first state of receiving the first query input of the first query type to a second state of receiving the second query input of the second query type, and to obtain a search result according to a query based on the first query input and the second query input.