Text-Augmented Object-Centric Relationship Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional scene graph generation models require fully annotated datasets for training, which are time-consuming and costly to obtain, limiting the number of object categories and decreasing accuracy for unseen categories during inference.
Innovation Solution
An image processing apparatus that combines fully grounded training samples with ungrounded caption samples, using a machine learning model with an image encoder, text encoder, multi-modal fusion encoder, and auto-regressive decoder to generate output text including relation-object pairs and locations, even without box annotations, through a two-step decoding process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If fully annotated datasets with box annotations are used for training, then the model can achieve accurate relation prediction, but the training process becomes time-consuming and costly
Solution Approach 1:
The model performs preliminary relation prediction without requiring box annotations during training. By using ungrounded caption samples that contain only text descriptions without spatial bounding boxes, the model learns relation predictions in advance without the time-consuming process of obtaining and processing fully annotated datasets with box annotations.
Solution Approach 2:
The patent extracts and removes the box annotation requirement from the training process. By separating the relation prediction task from the spatial localization task, the model can be trained on ungrounded data that excludes bounding box information, thereby eliminating the time-consuming annotation process while still achieving accurate relation prediction.
2Reliability
If fully annotated datasets with box annotations are used for training, then the model achieves good performance, but the cost of obtaining annotations increases
Solution Approach 1:
The patent extracts the relation prediction capability from the fully annotated dataset requirement. By using ungrounded caption samples that contain only text descriptions without bounding boxes, the model achieves reliable performance without incurring the high costs associated with obtaining and processing box annotations.
Solution Approach 2:
The patent uses inexpensive ungrounded caption samples as training data. These text-based samples without bounding boxes are much cheaper to obtain and process compared to fully annotated datasets, while still providing sufficient information for training the relation prediction model effectively.
3Device complexity
If the model is trained only on seen categories, then training is simpler, but the accuracy for unseen categories decreases
Solution Approach 1:
The model is trained to perform universal relation prediction that generalizes across both seen and unseen object categories. By using ungrounded caption samples with diverse text descriptions, the model learns category-agnostic relation patterns that can be applied to any object category, making the training process simpler while improving accuracy for unseen categories.
Solution Approach 2:
The patent changes the training parameter from category-specific to category-agnostic by removing bounding box information. This parameter change allows the model to learn general relation patterns that transfer across different object categories, thereby improving performance on unseen categories without increasing training complexity.
Data Source
AI summary
A method, apparatus, and non-transitory computer readable medium for image processing are described. Embodiments of the present disclosure obtain an image and an input text including a subject from the image and a location of the subject in the image. An image encoder encodes the image to obtain an image embedding. A text encoder encodes the input text to obtain a text embedding. An image processing apparatus based on the present disclosure generates an output text based on the image embedding and the text embedding. In some examples, the output text includes a relation of the subject to an object from the image and a location of the object in the image.