Weakly Supervised Referring Image Segmentation via Bounding Box Annotations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for training machine learning models for referring image segmentation are inefficient due to the tedious and time-consuming process of creating manually annotated segmentation masks, which often require large datasets and can include errors, making accurate segmentation challenging.
Innovation Solution
The method involves training a machine learning model using a training dataset with bounding box annotations instead of manually annotated segmentation masks, employing a loss function that includes multiple instance learning and conditional random field terms to generate segmentation masks based on text inputs, utilizing a text encoder, adaptor, image encoder, and transformer decoder architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manually annotated segmentation masks are used for training, then segmentation accuracy can be improved, but the annotation process becomes tedious and time-consuming
Solution Approach 1:
The patent introduces bounding box annotations as an intermediary training signal that is easier to obtain than pixel-level segmentation masks. The model uses these bounding boxes as weak supervision to learn segmentation without requiring time-consuming manual pixel-level annotations, thus reducing annotation time while maintaining segmentation capability
Solution Approach 2:
The patent creates pseudo-segmentation masks by copying and adapting bounding box annotations into pixel-level masks through algorithmic processing. This allows the model to learn from readily available bounding box data without requiring manual creation of detailed segmentation masks, significantly reducing annotation effort
2Reliability
If manually annotated segmentation masks are used for training, then model training can be performed, but a large number of annotations are required which are often not available
Solution Approach 1:
The patent replaces expensive, hard-to-obtain pixel-level segmentation masks with cheap, easily obtainable bounding box annotations. Bounding boxes require minimal annotation effort and can be obtained in large quantities, providing sufficient training data without the complexity of detailed mask creation
Solution Approach 2:
The patent makes the training system universal by using bounding box annotations that can be applied to any object in any image, rather than requiring task-specific pixel-level masks. This multi-functional annotation approach allows the same annotation format to work across diverse datasets and applications
3Manufacturing precision
If manually annotated segmentation masks are used, then training data can be created, but the masks can include errors and have poor quality
Solution Approach 1:
The patent uses bounding box annotations as an intermediary that is easier to create accurately than pixel-level masks. The simplified annotation task reduces human error while maintaining sufficient precision for training, as bounding boxes provide clear spatial boundaries without the complexity of pixel-level precision requirements
Data Source
AI summary
One embodiment of a method for training a machine learning model includes receiving a training data set that includes at least one image, text referring to at least one object included in the at least one image, and at least one bounding box annotation associated with the at least one object, and performing, based on the training data set, one or more operations to generate a trained machine learning model to segment images based on text, where the one or more operations to generate the trained machine learning model include minimizing a loss function that comprises at least one of a multiple instance learning loss term or an energy loss term