Multimodal Image Embedding Combining Visual and Textual Aspects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current recommendation systems for online services struggle to effectively incorporate both visual and textual aspects of images into their ranking algorithms, leading to suboptimal content item relevance predictions.
Innovation Solution
A multimodal machine learning approach that generates a single embedding for images by combining visual and textual aspects using an image captioning model and optical character recognition (OCR), allowing for the integration of both visual and textual information into the ranking process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If recommendation systems use only visual aspects of images, then the system complexity remains low, but the prediction accuracy of content item relevance is insufficient
Solution Approach 1:
The patent combines visual aspects (image embeddings from CNNs) and textual aspects (OCR text embeddings) into a unified multimodal embedding representation. This merging of multiple information sources enables more accurate relevance predictions while managing system complexity through integrated processing pipelines.
Solution Approach 2:
The recommendation system is enhanced to handle multiple types of data (visual and textual) from images through a unified multimodal framework. The system performs both image understanding and text recognition functions, making it multi-functional and adaptable to diverse content types without requiring separate specialized systems.
2Measurement precision
If recommendation systems incorporate both visual and textual aspects of images, then the prediction accuracy of content item relevance improves, but the system complexity increases
Solution Approach 1:
The patent segments the image processing into distinct functional modules: visual feature extraction using CNNs, textual aspect extraction using OCR, and embedding combination. This segmentation allows each component to be optimized independently while maintaining overall system accuracy, managing complexity through modular architecture.
Solution Approach 2:
The patent introduces embedding vectors as intermediary representations that bridge visual and textual aspects. These embeddings serve as a common language that allows the system to integrate diverse information types without direct complex interactions between visual processing and text processing components.
3Productivity
If manual hand-crafted features are used in ranking algorithms, then the interpretability of the system remains high, but the feature engineering process is time-consuming and labor-intensive
Solution Approach 1:
The patent replaces manual mechanical feature engineering with automated machine learning models. CNNs automatically learn visual features from images, and OCR automatically extracts textual aspects, eliminating the need for manual feature crafting while improving both efficiency and prediction accuracy.
Solution Approach 2:
The system employs representation learning where the models automatically discover and generate their own features from raw data. The CNNs and OCR components self-serve by extracting relevant features without human intervention, allowing the system to adapt to new content types without requiring manual feature engineering updates.
Data Source
AI summary
Described herein are systems and methods for generating an embedding—a learned representation—for an image. The embedding for the image is derived to capture visual aspects, as well as textual aspects, of the image. An encoder-decoder is trained to generate the visual representation of the image. An optical character recognition (OCR) algorithm is used to identify text/words in the image. From these words, an embedding is derived by performing an average pooling operation on pre-trained embeddings that map to the identified words. Finally, the embedding representing the visual aspects of the image is combined with the embedding representing the textual aspects of the image to generate a final embedding for the image.