Image Caption Generation Using Query Embeddings for Clean LMM Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large multimodal models (LMMs) face challenges in performance improvement due to noisy datasets generated through manual captioning or web collection, which often contain irrelevant text and errors, making it difficult to enhance their capabilities.
Innovation Solution
A method and system for automatically generating a high-quality training dataset by using web crawling to acquire images and text, processing them through an encoding model to generate query embeddings, and refining captions with a language model, followed by a filtering process to ensure data quality and relevance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual captioning is used to generate training data, then caption quality may be maintained, but substantial labor is required and human errors introduce noise
Solution Approach 1:
The system uses automated captioning models to generate captions independently without human intervention. The model processes images and generates captions automatically, eliminating the need for manual labor while maintaining consistent quality through algorithmic processing rather than human variability.
Solution Approach 2:
The patent replaces the mechanical human captioning process with an automated computational model. The captioning model uses machine learning algorithms to process images and generate captions, substituting human manual work with an automated system that can process large volumes of data efficiently and consistently.
2Productivity
If web collection is used to acquire training data, then data acquisition is efficient, but the acquired text and images contain noise and are difficult to use for training
Solution Approach 1:
The system extracts only the relevant and high-quality portions of web-collected data. The captioning model processes web-collected images and generates clean, relevant captions, separating the useful visual information from the noisy or irrelevant text that typically accompanies web images. This extraction process isolates the valuable image-caption pairs needed for training.
Solution Approach 2:
The captioning model acts as an intermediary between web-collected images and the training dataset. Instead of directly using web-collected text (which is noisy and irrelevant), the model generates new captions that serve as a bridge, translating image content into accurate, training-appropriate text descriptions that filter out web-specific noise.
3Quantity of substance
If web-collected text and images are used for training, then data volume is sufficient, but LMM performance improvement becomes difficult due to irrelevant text
Solution Approach 1:
The captioning model autonomously generates high-quality captions for web-collected images, creating a self-contained pipeline that produces training data with both sufficient volume and high relevance. The model processes large numbers of images efficiently, maintaining data quantity while ensuring each caption is specifically tailored to its corresponding image content.
Data Source
AI summary
There is provided a method for generating a caption, performed by a computing system. The method may comprise acquiring a first query embedding by inputting a first image and first text into an encoding model, wherein the encoding model is configured to output the first query embedding, in which features of at least one of the first image or the first text are reflected and acquiring a caption, in which features of the first image are reflected, by inputting the first query embedding and the first text into a language model.