AI Image Caption Enrichment Using Heat Map and Bounding Box Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image captioning systems using deep learning techniques struggle to accurately identify and name individuals in images, particularly when their faces are absent or not prominent in the training dataset, leading to errors and biases.
Innovation Solution
A method that utilizes a vision transformer model to generate a heat map representing the significance of image portions, combined with an expert recognition machine learning model to align bounding boxes with heat map portions, and modifies the AI-generated caption to include precise labels, leveraging explainable AI to enhance caption accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep learning techniques are used for image captioning, then the system can generate textual descriptions of images, but the system produces errors and biases when identifying individuals whose faces are absent or not prominent in the training dataset
Solution Approach 1:
The patent introduces heat maps as an intermediary visual representation that bridges the gap between the deep learning model's internal processing and the final caption output. The heat maps visually indicate which image regions influenced specific caption words, allowing the system to verify and correct identification accuracy for individuals whose faces may not be prominent in the training data
Solution Approach 2:
The system implements feedback by using the heat maps to monitor and evaluate the model's attention patterns during caption generation. When the heat map shows that a caption word refers to an individual but the corresponding image region does not clearly contain that individual's face, the system can detect and correct this discrepancy, improving reliability
2Reliability
If the model generates generic sentences to avoid errors, then biases are reduced, but the precision and usefulness of the captions decrease
Solution Approach 1:
The heat maps provide feedback that enables the system to distinguish between cases where generic descriptions are appropriate and cases where specific individual identification should be attempted. By monitoring attention patterns, the system can maintain precision when conditions are favorable while avoiding biases when conditions are not, rather than consistently using generic sentences
3Reliability
If multiple AI modalities and expert systems are integrated into the captioning system, then caption accuracy and reliability improve, but the system complexity increases
Solution Approach 1:
The patent segments the captioning system into distinct functional components: the deep learning captioning model, the heat map generation module, and the expert recognition system. This segmentation allows each component to specialize in its specific function while working together through well-defined interfaces, managing complexity through modular design
Data Source
AI summary
Systems and methods of the present disclosure enable enriching an artificial intelligence (AI)-generated caption including a textual description of an image. The image and the textual description is input into vision transformer model to produce heat map for the image, the heat map including a representation of a degree of significance of portion of the image to an identification of an item in the textual description based at least in part on the gradient. The image is input into an expert recognition machine learning model to output bounding box including label representative of the item. A spatial alignment within the image between the bounding box and the portion of the heat map is determined. The textual description of the AI-generated caption is modified to include the label of the item based on the spatial alignment within the image so as to produce a modified AI-generated caption associated with the item.