Object Detection via Attention Maps and Word Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Object detectors are limited to detecting only 'seen' classes in their training dataset and struggle to recognize 'unseen' classes, often biasing detection results and failing to detect regions of images for objects outside their training categories, with scaling the training dataset to include more classes being costly and time-consuming.
Innovation Solution
The use of a heterogeneous training approach where a tagging network generates an attention map and word embedding for a target concept, which are then provided as conditional inputs to a conditional detection network, allowing it to detect both seen and unseen classes without requiring extensive dataset scaling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the training dataset is scaled to include additional categories, then the detection capability for unseen classes is improved, but the cost and time required for annotation and manual effort increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a tagging network on a large-scale image tagging dataset with image-level annotations before using it to generate conditional inputs for the detection network. This pre-training phase prepares the system to handle unseen classes without requiring manual annotation of those specific classes, thereby resolving the contradiction between detection capability and annotation cost.
Solution Approach 2:
The patent introduces an intermediary tagging network that generates attention maps and word embeddings as conditional inputs to the detection network. This intermediary component enables the system to transfer knowledge from seen to unseen classes through learned representations rather than direct pixel-level annotations, reducing the need for expensive manual labeling while expanding detection capability.
2Measurement precision
If object detectors are trained with a large number of seen classes, then the detection accuracy for known objects is improved, but the ability to detect unseen classes deteriorates due to bias towards seen classes
Solution Approach 1:
The patent segments the detection system into two distinct networks: a tagging network trained on image-level annotations for learning general object concepts, and a detection network trained on pixel-level annotations for precise localization. This segmentation allows each network to specialize - the tagging network handles unseen class generalization while the detection network maintains high accuracy for seen classes, resolving the bias problem.
Solution Approach 2:
The patent changes the parameter representation by using word embeddings and attention maps as conditional inputs instead of direct class labels. This parameter transformation enables the detection network to adapt to unseen classes by conditioning on semantic representations generated by the tagging network, rather than being biased towards the finite set of seen classes in the training data.
3Adaptability or versatility
If zero-shot detectors are used to transfer knowledge from seen to unseen classes, then the classification capability for unseen classes is improved, but the object detection and region proposal capability deteriorates
Solution Approach 1:
The patent merges the strengths of zero-shot classification with pixel-level detection by combining the tagging network (which provides zero-shot classification capability) with the detection network (which provides accurate region proposals). The attention maps generated by the tagging network are integrated as conditional inputs to guide the detection network, enabling both unseen class classification and accurate region localization simultaneously.
Data Source
AI summary
In implementations of object detection in images, object detectors are trained using heterogeneous training datasets. A first training dataset is used to train an image tagging network to determine an attention map of an input image for a target concept. A second training dataset is used to train a conditional detection network that accepts as conditional inputs the attention map and a word embedding of the target concept. Despite the conditional detection network being trained with a training dataset having a small number of seen classes (e.g., classes in a training dataset), it generalizes to novel, unseen classes by concept conditioning, since the target concept propagates through the conditional detection network via the conditional inputs, thus influencing classification and region proposal. Hence, classes of objects that can be detected are expanded, without the need to scale training databases to include additional classes.