Image Entity Label Correction Using Multimodal Context Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current visual entity recognition models struggle with web-scale entity recognition, often hallucinating or outputting overly generic entities due to noisy datasets and ambiguity in language, and they typically focus on single entities per image, which is restrictive.
Innovation Solution
Implementing multimodal Large Language Models (LLMs) to verify and refine entity labels by accessing additional contextual information and generating rationales and question-answer pairs, using a high-quality, large-scale dataset curated from external knowledge sources like Wikipedia, to improve entity recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current visual entity recognition models are used, then processing speed is maintained, but entity recognition accuracy deteriorates due to hallucinations and generic outputs
Solution Approach 1:
The patent introduces an intermediary verification process using multiple machine-learned models that act as mediators between the initial entity recognition and the final output. These models verify candidate entity labels against contextual information from external data sources, preventing hallucinations and generic outputs while maintaining processing efficiency.
Solution Approach 2:
The system implements feedback mechanisms where the verification models provide corrective information back to the entity recognition process. When candidate labels are identified, the system retrieves contextual information and uses feedback from the verification models to confirm or correct labels, improving both accuracy and reliability.
2Measurement precision
If multiple verification models are implemented, then entity label accuracy is improved, but computing resource consumption increases
Solution Approach 1:
The system performs preliminary actions by pre-retrieving contextual information from external data sources before verification is needed. This preparation work is done in advance, so when verification occurs, the models can work more efficiently with pre-loaded contextual data, reducing peak computing resource consumption.
Solution Approach 2:
The verification process applies partial action by selectively verifying candidate entity labels based on their confidence scores and relevance. Not all candidates undergo full verification - only those that meet certain criteria are subjected to the complete verification process, reducing overall computing resource usage while maintaining accuracy for critical labels.
3Reliability
If contextual information from external sources is integrated, then entity recognition reliability is improved, but system complexity increases
Solution Approach 1:
The system achieves universality by designing machine-learned models that can handle multiple functions: initial entity recognition, candidate label generation, and verification against contextual information. This multi-functionality reduces the need for separate specialized components, managing system complexity while improving reliability through integrated verification.
4Productivity
If manual annotation is reduced, then productivity is improved, but annotation quality may deteriorate
Solution Approach 1:
The system implements self-service by enabling machine-learned models to automatically verify and correct entity labels without requiring manual annotation for every case. The models serve themselves by using contextual information from external sources to validate their own outputs, maintaining high annotation quality while dramatically improving productivity through automated verification.
Data Source
Figure 1A
Figure 1B
Figure 2A
AI summary
A computing device for recognizing an entity in an image includes one or more memories to store instructions and one or more processors to execute the instructions to perform operations, the operations including: receiving a request to identify an entity in an image, determining a candidate entity label for the entity based on the image, processing a plurality of inputs with one or more first machine-learned models to generate a corrected entity label for the candidate entity label, wherein the plurality of inputs include the image, a textual description of the image, and contextual information associated with the candidate entity label, and providing a first output including the corrected entity label for the entity in the image.