Open-Vocabulary Object Detector Training With Generated Region-Text Pairs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current state-of-the-art open-vocabulary object detection (OVD) models struggle with detecting novel object categories due to the lack of accurate region-level annotations and the dependency on pre-trained models, which creates a recursive dilemma of needing a good detector for generating accurate pseudo predictions.
Innovation Solution
A system and method that leverages generative models to synthesize region-text pairs using text-to-region and region-to-text processes, incorporating a scene-aware inpainting guider and a localization-aware region-text contrastive loss to enhance the training of OVD models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If web-crawled image-caption pairs are used for training, then the scale of training data is increased, but the accuracy of region-level annotations deteriorates
Solution Approach 1:
The patent uses a pre-trained OVD model to generate pseudo labels (copies of ground truth annotations) from web-crawled image-caption pairs. This allows the system to create synthetic region-level annotations at scale without manual annotation, resolving the contradiction between data quantity and annotation accuracy by using automated copying of detection results as training labels
Solution Approach 2:
The patent introduces a pre-trained OVD model as an intermediary between web-crawled image-caption pairs and the training process. This intermediary automatically generates region-level pseudo labels from captions, enabling the system to leverage large-scale web data while maintaining reasonable annotation quality through the mediation of the pre-trained detector
2Loss of information
If a pre-trained OVD model is used to generate pseudo predictions, then region-level annotations can be obtained, but the performance becomes dependent on the pre-trained model quality
Solution Approach 1:
The patent applies preliminary action by using a pre-trained OVD model to generate pseudo labels before the actual training process. This preliminary annotation step enables the system to create training data from web-crawled captions, and the recursive training process further improves the model using these pseudo labels, reducing dependency over time
Solution Approach 2:
The patent implements feedback through recursive training where the OVD model generates pseudo predictions that are used to train the model further. This feedback loop continuously improves the model's performance and reduces dependency on the initial pre-trained model quality, as each training iteration refines the pseudo label accuracy
3Quantity of substance
If captions are aligned with object proposals, then region-text pairs can be generated, but the alignment precision deteriorates due to lack of accurate image-caption correspondence
Solution Approach 1:
The patent uses the pre-trained OVD model to copy detection results and align them with caption phrases, generating region-text pairs automatically. This copying approach enables large-scale generation of aligned pairs without manual annotation, accepting some imprecision in exchange for scalability and data volume
Solution Approach 2:
The patent applies partial action by aligning only the most significant object proposals with captions, rather than attempting perfect alignment of all objects. This selective approach generates sufficient region-text pairs for training while acknowledging that not all alignments will be perfectly accurate, prioritizing quantity and coverage over exhaustive precision
Data Source
AI summary
Disclosed herein is a method of generating region-text pairs for training open-vocabulary object detection. The method innovates text-to-region and region-to-text processes, along with the introduction of a Scene-Aware Inpainting Guider and a Localization-Aware Region-Text Contrastive Loss.