Cross-Modal Video Captioning With Audio-Aware Transformer Fusion

The patent merges audio embeddings and video embeddings into a unified multi-modal embedding space using a cross-modal encoder. This allows the model to process both audio and video information simultaneously, enabling the generation of captions that include both speech content and audio elements like acoustic effects and speaker identity, while avoiding the need for separate processing pipelines that would increase complexity.

The cross-modal encoder is designed to handle multiple types of input data (audio embeddings and video embeddings) and generate a unified representation that captures both modalities. This multi-functional component replaces the need for separate processing systems, allowing the model to efficiently process diverse input types without proportionally increasing system complexity.

The patent implements a self-training mechanism where the model generates its own training data by producing captions for videos, which are then used to fine-tune and improve the model. This self-service approach eliminates the need for expensive manual annotation while continuously improving caption quality through iterative learning from generated examples.

The model uses parameter optimization techniques to adjust its internal representations and generation parameters during self-training. By changing parameters such as temperature, top-k sampling, and loss weights during different training stages, the model can generate diverse yet accurate captions that improve overall performance without requiring manual intervention.