VideoCoCa Model Zero-Shot Video-Text Transfer

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The computational resources required for training and fine-tuning machine learning models for video understanding tasks are substantial, especially when adapting models to new tasks or data types like video.

Innovation Solution

The proposed method leverages a pre-trained image-text processing model, specifically the Contrastive Captioners (CoCa) model, and adapts it for video understanding tasks by using pre-trained attentional pooling layers without additional training, thereby reducing computational resource consumption.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Use of energy by moving object

If a pre-trained image-text processing model is used for video understanding tasks, then computational resources are reduced, but the model requires adaptation to new data types

Engineering Contradiction:

Improvecomputational resourcesVSAvoidmodel adaptation capability

Core Design Contradiction:

Use of energy by moving objectVSAdaptability or versatility

Solution Approach 1:

The model performs preliminary action by pre-training on image-text pairs before being applied to video understanding tasks. The pre-trained attentional pooling layers and parameters are reused without additional training, enabling the model to adapt to video data types while minimizing computational resource consumption during deployment.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If additional parameters are added for new tasks, then task versatility is improved, but training computational resources increase

Engineering Contradiction:

Improvetask versatilityVSAvoidtraining computational resources

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The pre-trained image-text processing model is designed with universal attentional pooling layers that can process multiple data types including images and videos. The same model parameters and architecture are reused across different tasks (video classification, question answering, captioning) without requiring additional training, achieving multi-functionality while minimizing training computational resources.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If the model is trained from scratch for video tasks, then task performance is improved, but training time and resources increase

Engineering Contradiction:

Improvetask performanceVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250124708A1Video-text modeling with zero-shot transfer from contrastive captioners

Publication Date: 2025.04.17 GOOGLE LLC

AI summary

Provided is an efficient approach to establish a foundational video-text model for tasks including open-vocabulary video classification, text-to-video retrieval, video captioning and video question-answering. Some example implementations include a model which can be referred to as VideoCoCa. Example implementations reuse a pretrained image-text contrastive captioner (CoCa) model and adapt it to video-text tasks with little or minimal extra training. While previous works adapt image-text models with various cross-frame fusion modules (for example, cross-frame attention layer or perceiver resampler) and finetune the modified architecture on video-text data, aspects of the present disclosure leverage findings that the generative attentional pooling and contrastive attentional pooling layers in the image-text CoCa design are instantly adaptable to “flattened frame embeddings”, yielding a strong zero-shot transfer baseline for many video-text tasks.