Multimodal LLM Classification Using Text and Visual Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing classification systems rely solely on visual features extracted from images, limiting accuracy in zero-shot classification tasks.
Innovation Solution
A multimodal language model is used to generate both textual descriptions and class predictions of inputs, combining these with visual features to enhance classification accuracy by leveraging both modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If visual features are extracted from images using a pre-trained neural network, then classification can be performed, but classification accuracy is limited because visual features alone do not capture other descriptive features of the image
Solution Approach 1:
The patent combines visual features extracted from the image with text features generated by a language model describing the image. This merging of multiple feature types creates a more comprehensive representation that captures both visual and descriptive characteristics, thereby improving classification accuracy while reducing information loss.
Solution Approach 2:
The patent introduces a language model as an intermediary that generates textual descriptions of the image. This intermediary transforms visual information into text features, which are then combined with original visual features. The language model acts as a mediator that bridges the gap between raw visual data and classification requirements, capturing descriptive features that visual features alone miss.
2Measurement precision
If a multimodal language model generates both description and class prediction, then classification accuracy improves, but system complexity increases due to multiple processing steps
Solution Approach 1:
The patent employs a single multimodal language model that performs multiple functions: generating image descriptions, generating class predictions, and providing textual representations for classification. This multi-functional approach improves classification accuracy while managing system complexity by consolidating multiple processing capabilities into one unified model rather than requiring separate specialized components.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus for classification. In one aspect, a method includes receiving an input and a request to classify the input into one of a plurality of classes, processing the input using a multimodal model to generate (i) a description of the input and (ii) a class prediction, processing the description of the input and the class prediction using a text encoder embedding neural network to generate a (i) text description feature embedding and (ii) a prediction feature embedding, generating, from at least the description feature embedding and the prediction feature embedding, a query feature embedding representing the input, and classifying the input into one of the plurality of classes using the query embedding.