Geometric Vision Pre-Training with Multi-View 3D Cues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing unsupervised learning methods for computer vision, particularly in geometric tasks, are not well-suited for understanding 3D geometry, and supervised training with limited annotated data hinders effective performance in tasks like depth estimation, optical flow estimation, and pose estimation.
Innovation Solution
Adopting unsupervised pre-training methods that leverage multi-view geometry cues by using pairs of unannotated images from different viewpoints, employing pretext tasks such as cross-view completion and cross-view alignment to learn 3D geometry, followed by fine-tuning with annotated data for specific geometric vision tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised training with annotated data is used, then model performance on geometric vision tasks can be optimized, but the amount of available training data is limited due to significant annotation effort
Solution Approach 1:
The patent applies preliminary action by performing unsupervised pre-training on large unannotated datasets before supervised fine-tuning. The encoder is first trained without annotations using contrastive learning on image pairs, then fine-tuned with annotated data for the specific geometric vision task. This preliminary unsupervised training phase prepares the model to effectively leverage the limited annotated data afterward.
Solution Approach 2:
The patent uses contrastive learning as an intermediary approach between fully unsupervised and fully supervised training. By introducing a contrastive loss function that operates on unannotated image pairs, the method creates a middle ground that leverages large unannotated datasets while preparing the model for subsequent supervised fine-tuning on geometric vision tasks.
2Quantity of substance
If unsupervised training approaches are used, then large unannotated datasets can be utilized, but the approach is limited to tasks where optimal result can be determined without explicit annotation
Solution Approach 1:
The patent applies preliminary action by performing unsupervised pre-training on large unannotated datasets before supervised fine-tuning. The encoder is first trained without annotations using contrastive learning on image pairs, then fine-tuned with annotated data for the specific geometric vision task. This preliminary unsupervised training phase prepares the model to effectively leverage the limited annotated data afterward.
Solution Approach 2:
The patent uses contrastive learning as an intermediary approach between fully unsupervised and fully supervised training. By introducing a contrastive loss function that operates on unannotated image pairs, the method creates a middle ground that leverages large unannotated datasets while preparing the model for subsequent supervised fine-tuning on geometric vision tasks.
3Ease of manufacture
If existing unsupervised learning methods are used, then annotation effort is reduced, but these methods are not well-suited for understanding 3D geometry in geometric vision tasks
Solution Approach 1:
The patent applies local quality by designing a task-specific contrastive learning objective that focuses on geometric relationships. Instead of generic unsupervised learning, the method uses geometric augmentations (rotations, translations, perspective transformations) and geometric consistency constraints that are locally tailored to enhance 3D geometry understanding while remaining unsupervised.
Solution Approach 2:
The patent applies parameter changes by modifying the contrastive learning framework to incorporate geometric parameters. The method transforms images using geometric transformations (rotations, translations, perspective changes) and uses these transformed versions as positive pairs in the contrastive loss, thereby changing the learning parameters to focus on geometric structure rather than generic features.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems and computer-readable media for training a task specific machine learning model for a downstream geometric vision task are provided. Unsupervised pre-training of a pretext machine learning model is performed. The pretext machine learning model comprises an encoder having a set of encoder parameters and a decoder having a set of decoder parameters. The pre-training of the pretext machine learning model is performed based on pairs of unannotated images including a first image and a second image, wherein the first and second images depict the same scene, taken under different conditions or from different viewpoints. Following the pre-training, a task specific machine learning model for the downstream geometric vision task based on the pre-trained pretext machine learning model is constructed. The task specific machine learning model comprises a task specific encoder having a set of task specific encoder parameters. The set of task specific encoder parameters is initialized with the set of encoder parameters of the pre-trained pretext machine learning model. Subsequently, the task specific machine learning model is fine-tuned for the downstream geometric vision task.