Multimodal Query Embeddings for Zero-Shot Classification Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing classification systems rely solely on visual features extracted from images, limiting accuracy in zero-shot classification tasks where classes were not seen during training.
Innovation Solution
A multimodal language model is used to generate both a description and a class prediction of an input, which are then processed to create a query embedding for more accurate classification by combining textual and visual features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If visual features only are used for classification, then the system complexity is low, but the classification accuracy is limited
Solution Approach 1:
The patent combines visual features extracted from images with textual features generated by a language model to create a fused feature representation for classification. This merging of multiple feature types resolves the contradiction by improving classification accuracy through richer feature information while managing system complexity through integrated processing architecture.
Solution Approach 2:
The system employs a multimodal approach where a single classification framework handles both visual and textual inputs universally. The language model generates descriptive text that complements visual features, creating a universal feature fusion mechanism that improves accuracy across different classification tasks without requiring task-specific specialized systems.
2Adaptability or versatility
If specialized training data is used for each classification task, then the classification accuracy improves, but the adaptability decreases
Solution Approach 1:
The patent implements a universal classification framework that processes both visual and textual features through a shared architecture. The language model generates task-specific descriptive text from general input, enabling the system to adapt to different classification tasks without task-specific training data, thus maintaining both accuracy and versatility.
Solution Approach 2:
The system performs preliminary text generation by the language model before classification, creating descriptive representations of the input that capture task-relevant information. This preliminary action enables the classification module to work with enriched features that adapt to different tasks without requiring task-specific training, resolving the contradiction between accuracy and adaptability.
Data Source
AI summary
Methods, systems, and apparatus for classification. In one aspect, a method includes receiving an input and a request to classify the input into one of a plurality of classes, processing the input using a multimodal model to generate (i) a description of the input and (ii) a class prediction, processing the description of the input and the class prediction using a text encoder embedding neural network to generate a (i) text description feature embedding and (ii) a prediction feature embedding, generating, from at least the description feature embedding and the prediction feature embedding, a query feature embedding representing the input, and classifying the input into one of the plurality of classes using the query embedding.