Video Caption Decoding With Related-Video Memory Context
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video caption generating models focus excessively on the current video, leading to limited focus angles and reduced accuracy due to the reuse of similar but not identical decoded words across different videos, affecting the quality of generated captions.
Innovation Solution
A video caption generating model with an 'encoder + basic decoder + auxiliary decoder' structure, incorporating a memory mechanism that stores associations between candidate words and related videos, allowing it to consider similar visual features across multiple videos, thereby improving the accuracy of decoded words.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the video caption generating model focuses only on the current video, then the processing speed is improved, but the accuracy of decoded words deteriorates due to limited focus angles and reuse of similar decoded words across different videos
Solution Approach 1:
The model is segmented into two distinct decoders: a basic decoder that processes the current video independently, and an auxiliary decoder that processes related videos. This segmentation allows parallel processing of multiple video contexts without compromising speed, while improving accuracy by comparing decoded words across different video focuses
Solution Approach 2:
The auxiliary decoder is designed with multi-functionality to handle both current and related video processing. The memory structure stores decoded words from multiple video contexts, enabling the model to universally apply learned patterns across different videos while maintaining focus on the current video's specific requirements
2Measurement precision
If the model considers similar visual features across multiple videos, then the accuracy of decoded words is improved, but the device complexity increases due to the auxiliary decoder and memory structure
Solution Approach 1:
The memory structure is pre-populated with decoded words from related videos before the actual caption generation process. This preliminary action allows the auxiliary decoder to quickly access and compare against pre-processed information, reducing the computational complexity during real-time operation while maintaining high accuracy
Solution Approach 2:
The memory structure acts as an intermediary between the basic decoder and the auxiliary decoder. It stores and manages decoded words from multiple video contexts, facilitating efficient information exchange and comparison without requiring direct complex interactions between the two decoders, thus reducing overall system complexity
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
A video caption generation method comprises: encoding a target video by means of an encoder of a video caption generation model to determine a target visual feature of the target video; employing an attention mechanism to decode the target visual feature by means of a basic decoder of the video caption generation model, and determining first selection probabilities corresponding to respective candidate words; decoding the target visual feature by means of an auxiliary decoder of the video caption generation model to determine second selection probabilities corresponding to the respective candidate words, wherein a memory structure of the auxiliary decoder comprises reference visual context information corresponding to the respective candidate words, and the reference visual context information is generated according to relating videos corresponding to the candidate words; determining decoded words from the candidate words according to the first selection probabilities and the second selection probabilities; and generating a video caption according to the respective decoded words.