Image Caption Model Training Without Paired Annotations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image captioning technologies rely heavily on time- and labor-consuming manual annotation of image-sentence pairs for training, limiting the size and diversity of the training set and reducing accuracy.
Innovation Solution
A method for training an image caption model using an encoding CNN and decoding RNN, with adjustments based on matching and smoothness degrees, eliminating the need for paired image-sentence samples by incorporating a discriminative RNN and adjustment modules to optimize the decoding process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation of image-sentence pairs is used for training, then the model can be trained with paired data, but the process is time- and labor-consuming, limiting the size and diversity of the training set
Solution Approach 1:
The patent uses a pre-trained object recognition model to generate pseudo-labels (object detections) that copy the functionality of manual annotations. These generated labels serve as training data for the image captioning model, eliminating the need for time-consuming manual image-sentence pair annotation while providing sufficient training signals for accurate caption generation
Solution Approach 2:
The system performs self-annotation by using the object recognition model to automatically generate object detection results that serve as training labels. This self-service approach allows the training data to be generated automatically without external manual annotation, expanding the training set size and diversity while reducing preparation time
2Measurement precision
If manual annotation of image-sentence pairs is used for training, then the model can be trained with paired data, but the training set size and diversity are limited
Solution Approach 1:
The patent generates numerous pseudo-labeled training samples by applying the object recognition model to large collections of images. This copying approach creates abundant training data with diverse object compositions without requiring equivalent manual annotation efforts, thereby expanding the training set size and diversity while maintaining training quality
Solution Approach 2:
The patent performs preliminary object detection and labeling on training images using a pre-trained object recognition model before training the image captioning model. This preliminary action generates the necessary training labels automatically, enabling the use of large-scale image datasets without manual annotation and thus increasing the training set size and diversity
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Embodiments of this application disclose a method for training an image caption model, the image caption model including an encoding convolutional neural network (CNN) and a decoding recurrent neural network (RNN). The method includes: obtaining an image eigenvector of an image sample by using the encoding CNN; decoding the image eigenvector by using the decoding RNN, to obtain a sentence used for describing the image sample; determining a matching degree between the sentence obtained through decoding and the image sample, and adjusting the decoding RNN according to the matching degree; and determining a smoothness degree of the sentence obtained through decoding, and adjusting the decoding RNN according to the smoothness degree.