Image Captioning With Constrained Object-Guided Beam Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image captioning technologies struggle to accurately generate captions that include all relevant objects from an image while excluding irrelevant ones, and often fail to maintain semantic coherence between the image and the generated text.
Innovation Solution
A method and apparatus that utilize a two-layer LSTM with an area-level attention mechanism, combined with a beam search and adversarial training, to generate captions by constraining the inclusion of identified objects and excluding irrelevant ones, using a semantic correlation between image triplets and generated sentences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large datasets of paired images and sentences are used for training, then the captioning device can achieve better training performance, but the requirement for data preparation and storage increases significantly
Solution Approach 1:
The system performs self-service by automatically generating pseudo sentence descriptions from images using the captioning device itself, eliminating the need for external paired datasets. The generated pseudo sentences are then used to train and retrain the captioning device, creating a self-sufficient training loop that reduces dependency on large pre-prepared datasets.
Solution Approach 2:
The system creates copies of image data by generating multiple pseudo sentence descriptions for the same image through iterative training. These pseudo sentence copies serve as training samples, allowing the system to expand its training data without requiring additional paired image-sentence datasets.
2Productivity
If traditional captioning methods are used, then the system can process images efficiently, but the accuracy of caption generation is insufficient
Solution Approach 1:
The system implements feedback by using the captioning device to generate pseudo sentences, then using these pseudo sentences to retrain and improve the device. This iterative feedback loop continuously enhances caption accuracy while maintaining processing efficiency, as the system learns from its own generated outputs without requiring external validation data.
3Ease of manufacture
If paired datasets are required for training, then the training process is straightforward, but the system cannot handle unpaired data scenarios
Solution Approach 1:
The system achieves universality by designing a training framework that can handle both paired and unpaired data scenarios. The pseudo sentence generation mechanism allows the captioning device to train effectively regardless of whether paired datasets are available, making the system adaptable to various data formats and scenarios while maintaining training simplicity.
Data Source
AI summary
A method and apparatus for generating a captioning device, and a method and apparatus for outputting a caption. The method for generating a captioning device comprises: acquiring a sample image set; inputting the sample image set into an image encoder of a sentence generator, so as to output an object set; grouping the object set into a first object set and a second object set, wherein the first object set is an object set that is included within a preset object set, and the second object set is an object set that is excluded from the preset object set; inputting, into a sentence decoder of the sentence generator, the object set output by the image encoder, and performing a beam search in a decoding step by taking the first object set and the second object set as constraint conditions, so as to generate a pseudo-image sentence pair set; and training the sentence generator by taking the pseudo-image sentence pair set as a sample set, so as to obtain a captioning device.