Image-Text Query Fusion for More Accurate Object Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query services often fail to meet user expectations due to the limitations of image-based or text-based methods, resulting in less comprehensive and accurate query outcomes, particularly in scenarios with diverse user needs.
Innovation Solution
An information processing method that integrates image-text query information, determines the information attribute type, constructs image-text fusion information, and performs object retrieval and ranking to align with user intent, using multi-modal models for enhanced accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image-based or text-based query methods are used separately, then the query process is simple, but the query results are not comprehensive and accurate enough
Solution Approach 1:
The patent combines image-based query and text-based query into a unified multi-modal query system. The system simultaneously processes both image and text inputs, fuses their respective features, and performs joint retrieval to produce comprehensive query results that leverage the strengths of both modalities, thereby improving accuracy without requiring completely separate systems
Solution Approach 2:
The patent creates a universal query system that can handle multiple query types (image-only, text-only, and image-text combined) through a single multi-modal framework. The system dynamically adapts to different input combinations and applies appropriate fusion strategies, making it versatile across various query scenarios while maintaining a unified architecture
2Adaptability or versatility
If single-modality query (image or text only) is used, then the system complexity is low, but the information coverage is narrow
Solution Approach 1:
The patent implements a dynamic query system that adapts its processing based on the input modality combination. The system dynamically selects fusion strategies (early fusion, late fusion, or hybrid) depending on whether the input is image-only, text-only, or combined, allowing it to flexibly adjust its complexity to match the query requirements while maximizing information coverage
Solution Approach 2:
The patent segments the query processing into distinct modular components: image feature extraction module, text feature extraction module, feature fusion module, and retrieval module. This segmentation allows each component to specialize in specific tasks while maintaining overall system versatility, enabling the system to handle different query types through coordinated module operations
3Measurement precision
If multi-modal image-text fusion is implemented, then query accuracy improves, but processing complexity increases
Solution Approach 1:
The patent applies partial fusion action by selectively fusing image and text features only when both modalities are provided and beneficial. The system can choose to use only image features, only text features, or fuse both depending on the query context, avoiding unnecessary processing complexity while maintaining high precision when fusion is truly advantageous
Solution Approach 2:
The patent introduces a feature fusion module as an intermediary between image and text processing pipelines. This mediator component responsibly combines features from both modalities using strategies such as concatenation, attention mechanisms, or hybrid approaches, managing the complexity of multi-modal integration in a controlled and systematic manner
Data Source
Figure 1
Figure 2
Figure 3(a)~3(b)
AI summary
The embodiments of the present disclosure provide an information processing method and apparatus, as well as a product query method and apparatus. The information processing method includes: obtaining image-text query information comprising image query information and text query information, and determining an information attribute type corresponding to the image-text query information; identifying the image query information and the text query information within the image-text query information, and constructing image-text fusion information based on the image query information and the text query information; performing object retrieval for the image query information, the text query information, and the image-text fusion information respectively to obtain an image-retrieved object, a text-retrieved object, and an image-text retrieved object; ranking the image-retrieved object, the text-retrieved object, and the image-text retrieved object according to the information attribute type, and determining a target object corresponding to the image-text query information based on a ranking result.