Multi-Modal Image Classification Using Embedded Text Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image classifiers rely on additional descriptive text to accurately classify images, leading to misclassifications when such text is absent, and inefficient use of resources in reclassifying images.
Innovation Solution
Implementing a system that extracts and analyzes embedded text characters from images using machine learning models to generate text vectors and image vectors, which are then processed to create multi-modal vectors for accurate image classification, reducing reliance on external text descriptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional image classifiers use additional descriptive text for classification, then classification accuracy is improved, but system complexity and dependency on external data increase
Solution Approach 1:
The patent extracts text information directly from the image itself using optical character recognition (OCR) technology, eliminating the need for external descriptive text files. The system extracts embedded text characters from the image and processes them through machine learning models to generate text vectors, which are then combined with image vectors to create multi-modal vectors for classification.
Solution Approach 2:
The patent creates a universal classification system that can process images with or without embedded text. The multi-modal vector approach allows the same classification model to handle both text-containing and text-free images uniformly, making the system more versatile and reducing dependency on external data sources.
2Productivity
If conventional image classifiers rely on external text descriptions, then classification can be performed, but misclassifications occur when text is absent
Solution Approach 1:
The patent enables the image itself to provide the necessary text information through embedded OCR extraction. The image serves its own classification needs by extracting and processing its own embedded text characters, eliminating the need for external text descriptions and ensuring consistent performance whether text is present or absent.
Solution Approach 2:
The system performs preliminary OCR extraction and text vector generation during the initial image processing stage. By preparing text representations in advance as part of the standard classification pipeline, the system ensures that text information is ready for classification without requiring separate text processing steps or external data inputs.
3Object-affected harmful factors
If conventional image classifiers filter images based on classification, then unwanted content is removed, but processing resources are wasted on reclassification
Solution Approach 1:
The patent replaces conventional mechanical filtering approaches with an intelligent multi-modal classification system. By using combined text and image vectors to create comprehensive multi-modal representations, the system achieves more accurate initial classifications that reduce the need for subsequent reprocessing and manual review of filtered content.
Data Source
AI summary
Technologies for generating a multi-modal representation of an image based on the image content are provided. The disclosed techniques include receiving an image, to be classified, that comprises one or more embedded text characters. The one or more embedded text characters are identified from the image and a first machine learning model is used to generate a text vector that represents a numerical representation of the one or more embedded text characters. A second machine learning model is used to generate an image vector that represents a numerical representation of the graphical portion of the image. The text vector and the image vector are used as input to generate a multi-modal vector that contains information from both the text vector and the image vector. The image may be classified into one of a plurality of image classifications based upon the information in the multi-modal vector.