Multimodal Object Detection Using Visual and Textual Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image processing techniques fail to accurately identify specific objects in scenes with many similar objects, especially when objects are oriented in ways that obscure visual features, leading to incorrect classification and difficulty in locating desired items in real-world environments.
Innovation Solution
A multimodal approach combining object detection using neural networks with textual analysis, including salient feature detection and optical character recognition, to identify and highlight target objects within a captured video scene, utilizing augmented reality for rapid location.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current image-based object identification techniques are used, then the system can identify objects in a scene, but it cannot discern target objects from similar-looking objects or provide specific classification (e.g., cannot distinguish between different flavored soup cans)
Solution Approach 1:
The patent segments the object identification task into multiple independent components: visual feature extraction (color, shape, size), textual information extraction (product labels, branding), and spatial relationship analysis. Each component processes specific aspects of the object separately, allowing the system to distinguish between similar objects by combining results from multiple segmentation points rather than relying on a single broad classification.
Solution Approach 2:
The patent adds new dimensions to object identification by incorporating textual information from product labels and packaging, which was not present in traditional image-only analysis. This dimensional expansion from purely visual features to multimodal features (visual + textual + spatial) enables the system to distinguish between objects that appear similar in 2D images but have different textual identifiers.
2Reliability
If current object detection techniques are used, then the system can identify objects in a scene, but it fails when objects are oriented or occluded to show only partial view
Solution Approach 1:
The patent extracts and stores key visual features (color, shape, size) and textual information from objects during an initial phase, creating a feature database before the actual detection task. This preliminary extraction allows the system to recognize objects even when only partially visible during operation, as the complete feature set is pre-available for comparison against partial observations.
Solution Approach 2:
The patent uses feature descriptors and textual label matching as intermediary representations that bridge the gap between complete object views and partial views. These intermediaries allow the system to infer object identity from incomplete visual information by comparing partial observations against pre-extracted feature representations and textual data.
3Productivity
If broader classes are used in training, then the system can quickly identify objects, but it cannot provide specific labels (e.g., labels all cans as food cans instead of specific soup types)
Solution Approach 1:
The patent segments the classification process into hierarchical levels: first identifying the broad object category (e.g., canned food), then extracting specific features (color, shape, size), and finally matching textual information from labels to provide specific classification (e.g., chicken noodle soup). This segmentation allows the system to maintain speed through hierarchical processing while achieving high specificity through multi-stage analysis.
Solution Approach 2:
The patent transitions from single-stage classification to multi-stage classification by adding textual information analysis as an additional dimension. This multimodal approach enables the system to first perform quick visual classification and then refine it with textual label matching, achieving both speed and specificity through sequential processing of different information dimensions.
Data Source
AI summary
Techniques are provided for identifying objects (such as products within a physical store) within a captured video scene and indicating which of object in the captured scene matches a desired object requested by a user. The matching object is then displayed in an accentuated manner to the user in real-time (via augmented reality). Object identification is carried out via a multimodal methodology. Objects within the captured video scene are identified using a neural network trained to identify different types of objects. The identified objects can then be compared against a database of pre-stored images of the desired product to determine if a close match is found. Additionally, text on the identified objects is analyzed and compared to the text of the desired object. Based on either or both identification methods, the desired object is indicated to the user on their display, via an augmented reality graphic.