Open Vocabulary Object Detection via Pseudo Bounding-Box Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep object detection methods rely heavily on human-provided bounding-box annotations, which are costly and time-consuming, limiting their ability to detect a small number of object categories and hindering the detection of novel classes without extensive human labeling.
Innovation Solution
The approach generates pseudo bounding-box labels automatically using pre-trained vision-language models and large-scale image-caption datasets, allowing for open vocabulary object detection without human-provided annotations by computing activation maps and converting them into bounding-box labels for object categories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human-provided bounding-box annotations are used for training, then detection performance on pre-defined categories is improved, but labeling cost and time consumption increase significantly
Solution Approach 1:
The system uses pre-trained vision-language models to automatically generate pseudo bounding-box annotations from image-caption pairs without human intervention. The model extracts object mentions from captions and generates corresponding bounding boxes automatically, enabling the system to create its own training data without external human labeling
Solution Approach 2:
The system creates pseudo bounding-box annotations by copying and adapting the localization capabilities of pre-trained vision-language models. Instead of manually creating annotations, it replicates the annotation process through automated model inference on large-scale image-caption datasets
2Measurement precision
If human-provided bounding-box annotations are used, then training data quality is improved, but the number of detectable object categories remains limited
Solution Approach 1:
The system uses a universal pre-trained vision-language model that can handle multiple object categories across different domains. The model's language understanding capability allows it to process captions mentioning diverse object types, enabling detection across many categories without requiring separate training for each category
Solution Approach 2:
The system performs preliminary training on a large number of categories using automatically generated pseudo annotations before fine-tuning on specific target categories. This preliminary exposure to diverse categories enables the model to generalize better to novel classes
3Adaptability or versatility
If extensive human labeling is performed to detect novel classes, then detection coverage is improved, but cost and time requirements increase
Solution Approach 1:
The system automatically generates training data for novel categories by leveraging pre-trained vision-language models and external knowledge sources. The model self-supervises itself by extracting object information from captions and generating corresponding annotations without human intervention
Solution Approach 2:
The system uses pre-trained vision-language models as intermediaries to bridge the gap between unannotated images and structured bounding-box annotations. These models translate natural language captions into localized object representations, enabling automated annotation generation
Data Source
AI summary
Embodiments described herein provide methods and systems for open vocabulary object detection of images. given a pre-trained vision-language model and an image-caption pair, an activation map may be computed in the image that corresponds to an object of interest mentioned in the caption. The activation map is then converted into a pseudo bounding-box label for the corresponding object category. The open vocabulary detector is then directly supervised by these pseudo box-labels, which enables training object detectors with no human-provided bounding-box annotations.