Training vision models with unified contrastive learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision models lack broad general transferability and adaptability, requiring significant customization and supervised training, limiting their applicability to diverse real-world tasks.
Innovation Solution
A computer vision foundation model is pre-trained using a hierarchical vision transformer with shifted windows and convolutional embedding, leveraging unified image-text contrastive learning on weakly labeled web-scale data, enabling scalable and transferable adaptability across various tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing computer vision models are trained using exhaustive sets of training data with supervision, then the models can perform specific tasks with high accuracy, but the models lack broad general transferability and require significant customization for different tasks
Solution Approach 1:
The patent applies universality by training a single foundation model on diverse, multi-domain data (images, text, audio, video) to enable it to perform multiple different computer vision tasks without task-specific customization. The model is designed to be universally applicable across classification, retrieval, segmentation, and other vision tasks through unified contrastive learning that captures general visual concepts rather than task-specific patterns
Solution Approach 2:
The patent applies preliminary action by pre-training the foundation model on large-scale diverse data before deployment to specific tasks. This pre-training phase establishes broad general transferability upfront, allowing the model to be quickly adapted to new tasks with minimal additional training rather than requiring exhaustive task-specific training from scratch
2Adaptability or versatility
If computer vision models are trained on large-scale diverse data sets, then the models can be adapted to a wide range of downstream tasks, but the training requires significant computing resources and time
Solution Approach 1:
The patent applies continuity of useful action through contrastive learning that continuously refines the foundation model's understanding of visual concepts across diverse data. The unified contrastive learning framework processes multiple data types (images, text, audio, video) in an integrated manner, maximizing the useful learning from each training example and reducing the total computational resources needed compared to separate training processes
Solution Approach 2:
The patent introduces contrastive learning as an intermediary mechanism that efficiently bridges the gap between diverse training data and task-specific performance. By using contrastive objectives that learn general visual representations without task-specific supervision, the system reduces computational overhead while maintaining adaptability to downstream tasks
3Ease of operation
If models are trained to perform zero-shot learning tasks, then the models can handle unseen tasks without additional training, but the models can only be transferred to related computer vision schemes with limited general transferability
Solution Approach 1:
The patent extends zero-shot capability from traditional classification tasks to multiple dimensions including image retrieval, segmentation, and other vision tasks by training on diverse data types and formats. This multi-dimensional training approach enables the model to generalize across different task types rather than being limited to related schemes within a single task dimension
Data Source
AI summary
Examples are provided for pre-training a computer vision foundation model. A representative method comprises curating a pre-training database of image-text pairs from weakly labeled data. Language is encoded of text descriptions from the image-text pairs. The images of the image-text pairs are encoded using a hierarchical vision transformer with shifted windows and convolutional embedding. Based on the encoded images and the encoded language, the computer vision foundation model is pre-trained via unified image-text contrastive learning.