Multimodal OOD Detection Using Masked Image Features and Text Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing OOD detection methods, particularly single-modal and multi-modal approaches, struggle to effectively distinguish between in-distribution and out-of-distribution data due to reliance on visual features alone or insufficient incorporation of textual representations, leading to misclassifications and limited generalization.
Innovation Solution
A multi-modal OOD detection system utilizing a general feature representation and informative textual content with a hierarchical structure, employing a general feature index to mask class-specific features and incorporate superclass prompts, calculates an OOD score through similarity analysis of masked image features and text embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single-modal learning approaches are used that depend only on visual data for OOD detection, then the system is simpler to implement, but the detection accuracy is limited when OOD inputs visually resemble ID data but exhibit semantic distinctions
Solution Approach 1:
The patent combines visual data and textual data into a unified multi-modal learning framework. The system processes both image features and text features simultaneously, merging them into a joint feature space for OOD detection. This allows the system to leverage both visual similarity and semantic meaning, resolving the limitation of single-modal approaches where OOD inputs could visually resemble ID data but differ semantically.
Solution Approach 2:
The patent introduces text embeddings as an intermediary modality that bridges visual features and semantic information. By encoding textual descriptions into embeddings and comparing them with image features, the system creates a mediator layer that captures semantic distinctions beyond visual appearance, improving OOD detection accuracy without excessive complexity.
2Use of energy by moving object
If traditional single-modality methods rely on robust feature and specific distance metrics to discern OOD data from ID data, then the method is computationally efficient, but it overlooks semantic information embedded in textual descriptions
Solution Approach 1:
The patent segments the feature extraction process into distinct visual and textual components. Each modality is processed separately through its own encoder, allowing efficient computation for each stream while preserving their unique information. The visual encoder processes images and the text encoder processes descriptions independently, then their features are combined, maintaining computational efficiency while preventing information loss.
Solution Approach 2:
The patent adds a textual dimension to the traditional visual-only approach. By introducing text embeddings as an additional feature dimension, the system captures semantic information that exists in a different dimensional space from visual features. This multi-dimensional feature space allows the system to retain computational efficiency while incorporating previously overlooked semantic information.
3Productivity
If OOD detection methods use only visual features, then the processing is faster and requires less data, but the system cannot effectively identify semantic distinctions between OOD and ID data
Solution Approach 1:
The patent performs preliminary encoding of both visual and textual data into compact feature representations before the actual OOD detection comparison. By pre-processing images and text into efficient embeddings, the system maintains fast processing speed while ensuring that rich semantic information is captured in the feature vectors, which then can be quickly compared for reliable OOD detection.
Solution Approach 2:
The patent creates a composite feature representation that combines visual and textual characteristics. Similar to composite materials in engineering, this composite feature space integrates the strengths of both modalities - the visual fidelity and the semantic richness - resulting in a more reliable OOD detection system that maintains processing efficiency through optimized feature fusion.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method includes encoding a set of hierarchical text prompts to define a set of text embeddings, where the set of hierarchical text prompt defines a primary informative prompt and a secondary informative prompt associated with the primary informative prompt. The method further includes encoding an input image to define a plurality of feature representations, changing a value of one or more identified feature representations among the plurality of feature representations to mask the one or more identified feature representation and define a general feature representation of the input image based on a class-specific threshold indicative of boundary between a class-specific feature and a general feature. The method further includes classifying the input image based on an out-of-distribution (OOD) score determined using a similarity analysis of the general feature representation and the set of text embeddings.