Multimodal Entity Label Correction for Web-Scale Image Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current models for visual entity recognition struggle with web-scale entity identification, often hallucinating or outputting overly generic entities due to noisy datasets and ambiguous language, and typically focus on single entities per image, limiting their effectiveness in complex scenarios.
Innovation Solution
Implementing multimodal Large Language Models (LLMs) to verify and refine entity labels by accessing contextual information from external sources like Wikipedia, generating rationales, and creating question-answer pairs, thereby improving dataset quality and entity recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current models use noisy datasets and standard training methods, then training process is simple, but entity recognition accuracy deteriorates with hallucinations and generic outputs
Solution Approach 1:
The patent applies preliminary action by using LLMs to pre-generate high-quality training data and rationales before the actual model training. The LLMs anticipate and prevent hallucinations by generating corrected entity labels and explanatory rationales in advance, which then guide the training process to achieve accurate entity recognition without requiring complex real-time correction mechanisms.
Solution Approach 2:
The patent implements feedback by using LLMs to evaluate candidate entity labels and generate corrected versions based on contextual information from images and external sources. This feedback loop continuously refines the training data quality, allowing the model to learn from corrected examples and improve its entity recognition accuracy iteratively.
2Adaptability or versatility
If models focus on single entities per image, then processing is simple, but effectiveness in complex scenarios deteriorates
Solution Approach 1:
The patent applies universality by training the model to handle multiple entity recognition tasks simultaneously using a unified architecture. The model can identify single entities or multiple entities in an image, generate rationales, and answer questions about entities, all through the same trained system. This multi-functional capability enables the model to adapt to complex scenarios with multiple entities without requiring separate specialized models.
3Productivity
If manual annotation is used for training data, then data quality is high, but time consumption and cost increase
Solution Approach 1:
The patent applies self-service by using LLMs to automatically generate, verify, and correct training data without human intervention. The LLMs autonomously evaluate candidate entity labels against contextual information from images and external knowledge sources, generate rationales for their corrections, and produce high-quality training datasets independently. This self-service approach eliminates the need for manual annotation while maintaining or improving data quality through the LLMs' sophisticated understanding of context and entity relationships.
Data Source
AI summary
A computing device for recognizing an entity in an image includes one or more memories to store instructions and one or more processors to execute the instructions to perform operations, the operations including: receiving a request to identify an entity in an image, determining a candidate entity label for the entity based on the image, processing a plurality of inputs with one or more first machine-learned models to generate a corrected entity label for the candidate entity label, wherein the plurality of inputs include the image, a textual description of the image, and contextual information associated with the candidate entity label, and providing a first output including the corrected entity label for the entity in the image.