Image Caption Quality Evaluation Using Contextual Vector Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image caption generators often misidentify objects in images, leading to inaccurate caption content, which can mislead users about the image's representation, making generated captions unreliable as substitutes for image content.
Innovation Solution
A system that evaluates generated captions by comparing them to contextual data associated with the image, using machine-learned models to generate confidence scores based on vector representations of contextual and caption data, allowing for the selection and presentation of the most accurate caption, independent of the caption generator's source.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional image caption generators are used to generate captions based on image content, then captions can be provided for users who cannot view images, but the caption accuracy deteriorates due to object misidentification
Solution Approach 1:
The patent introduces an intermediary evaluation system that acts as a mediator between the image caption generator and the final caption output. This evaluation system uses multiple machine-learned models to assess caption quality by comparing generated captions against contextual data from the image, thereby improving accuracy without compromising accessibility
Solution Approach 2:
The patent replaces the conventional single-stage caption generation mechanism with a multi-stage system incorporating multiple machine-learned evaluation models. These models substitute simple object recognition with sophisticated contextual analysis, replacing basic caption generation with evaluated and verified caption output
2Measurement precision
If multiple machine-learned models are used to evaluate caption quality, then caption accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by extracting contextual data from images (objects, categories, hashtags, OCR text) before caption generation and evaluation. This pre-processing of image context enables the multiple evaluation models to work more efficiently by having relevant reference data ready, reducing the overall processing time despite using multiple models
Solution Approach 2:
The patent segments the caption evaluation process into multiple independent machine-learned models, each evaluating different aspects of caption quality (object identification accuracy, contextual relevance, etc.). This segmentation allows parallel processing of different evaluation tasks, managing computational complexity while maintaining comprehensive accuracy assessment
Data Source
AI summary
Technologies for evaluating, scoring, and determining whether to present a caption of an image are provided. The disclosed techniques include receiving an image with associated metadata. Contextual data is identified from the image and the metadata. A generated caption for the image is received from an image caption generator. A first vector representation is generated based on the contextual image data and a second vector representation is generated based on the generated caption. A machine learned model generates a score for the generated caption using the first vector representation and the second vector representation. The score represents a confidence value defining how accurately the caption describes the image. Based on the score, the caption may be presented along with the image on a client device.