Object and Action Recognition via Relative Text-Image Distance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional object and action recognition methods face low accuracy due to the inherent disconnect between textual and visual representations, and the high costs and time required for data gathering, annotation, and training of large language models (LLMs).
Innovation Solution
A relative vector distance process is used to classify images and text by comparing multiple classes of text to an image, selecting the text with the smallest vector distance relative to other classes, eliminating the need for new dataset collection and LLM fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional cosine similarity comparison between textual descriptions and visual data is used, then the process is simple and fast, but the classification accuracy is low due to inherent differences between text and visual representations
Solution Approach 1:
The patent introduces a relative vector distance computation as an intermediary mechanism between image vectors and text vectors. Instead of directly comparing text and visual representations, the system computes vector distances between the image vector and multiple text vectors, then uses relative comparison to determine classification. This intermediary process bridges the gap between different modalities (text and visual) and resolves the accuracy issue while maintaining computational feasibility
Solution Approach 2:
The patent changes the parameter of comparison from direct cosine similarity to relative vector distance. By computing the vector distance between an image vector and multiple text vectors, and then comparing these distances relatively (determining which text vector has the smallest distance), the system transforms the classification approach to achieve higher accuracy without requiring complex model fine-tuning
2Measurement precision
If a large language model is fine-tuned with annotated datasets to improve accuracy, then classification performance improves, but the cost of data gathering, annotation, and training increases significantly
Solution Approach 1:
The patent uses vector representations (embeddings) as copies or proxies of the actual data. Instead of requiring annotated datasets and fine-tuning the LLM, the system computes vector distances between image vectors and text vectors. These vectors serve as compressed representations that capture the essential features needed for classification, eliminating the need for expensive data collection and annotation processes while maintaining high accuracy
Solution Approach 2:
The patent changes the approach from model fine-tuning (modifying the LLM parameters) to vector distance computation (comparing pre-computed embeddings). By using relative vector distance comparison, the system achieves accurate classification without the need to gather annotated datasets or train/fine-tune the LLM, thereby significantly reducing resource requirements
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Example implementations include a method, apparatus and computer-readable medium of object/action recognition using a text-based classification model, comprising generating an image vector configured to represent one or more features of the first image. Additionally, the implementations further include computing a vector distance between the image vector and each of a first text vector and a second text vector, wherein the first text vector is configured to represent a first text, and wherein the second text vector is configured to represent a second text. Additionally, the implementations further include classifying the first image according to the first text or the second text based on which computed vector distance indicates a highest similarity between the image vector and either the first text vector or the second text vector relative to the other of the first text vector or the second text vector.