Weakly-supervised referring expression segmentation

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Conventional systems for segmenting objects in digital images referred to by text suffer from computational inefficiencies, inaccuracies, and operational inflexibilities due to the need for large-scale mask annotations, which consume significant resources and fail to accurately identify target objects, especially in zero-shot scenarios.

Innovation Solution

A weakly-supervised referring expression segmentation pipeline that augments training sets with inferred masks, utilizes cross-attention-based feature fusion, and implements attention mask consistency regularization to enhance spatial localization, allowing models to generate accurate segmentation masks with minimal annotations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If large-scale mask annotations are used to train text and image models, then segmentation accuracy is improved, but computational resources and time consumption increase significantly

Engineering Contradiction:
Improvesegmentation accuracyVSAvoidcomputational resources
Core Design Contradiction:
Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The system performs self-improving capabilities to generate mask labels automatically. The pipeline includes a pseudo-labeling mechanism where the model generates its own training data by inferring mask labels from unlabeled images, eliminating the need for manual annotation of all training samples. This self-service approach significantly reduces computational resources while maintaining segmentation accuracy.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The system creates synthetic training data by copying and adapting from limited annotated samples. Through pseudo-labeling, the model generates artificial mask labels that replicate the structure and quality of ground truth annotations, allowing the system to train on much larger datasets without proportionally increasing annotation costs.

Inventive Principle:
Principle #26Copying

2Measurement precision

If abundant ground-truth mask annotations are used for training, then model accuracy is improved, but the complexity and cost of data preparation increases

Engineering Contradiction:
Improvemodel accuracyVSAvoiddata preparation complexity
Core Design Contradiction:
Measurement precisionVSDevice complexity

Solution Approach 1:

The system automatically generates its own training data through pseudo-labeling. The pipeline takes limited annotated samples and automatically creates extensive training datasets by inferring mask labels from unlabeled images, eliminating the complex manual process of preparing large-scale annotations while maintaining high model accuracy.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The system performs preliminary pseudo-labeling of unlabeled samples before actual training. This preliminary action creates a large pool of pre-processed training data with inferred masks, which then serves as the foundation for final model training, significantly simplifying the overall data preparation process.

Inventive Principle:
Principle #10Preliminary action

3Measurement precision

If traditional supervised learning with full annotations is used, then segmentation performance is improved, but adaptability to zero-shot scenarios deteriorates

Engineering Contradiction:
Improvesegmentation performanceVSAvoidzero-shot capability
Core Design Contradiction:
Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The system generates pseudo-labels for unlabeled images, enabling it to learn from data it has never seen before during training. This self-improving capability allows the model to generalize to zero-shot scenarios where the target object or category was not present in the original training annotations, significantly enhancing adaptability while maintaining segmentation performance.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The pipeline creates a universal training approach that works for both labeled and unlabeled data, as well as for known and unknown categories. The pseudo-labeling mechanism provides a unified framework that enables the same training process to handle diverse scenarios including zero-shot segmentation, making the system highly adaptable across different tasks.

Inventive Principle:
Principle #6Universality (Multi-functionality)

4Measurement precision

If manual mask annotation is performed for all training samples, then training data quality is improved, but time consumption and labor requirements increase

Engineering Contradiction:
Improvetraining data qualityVSAvoidannotation time
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The system performs automatic pseudo-labeling to generate training data quality comparable to manual annotations. The pipeline includes a filter mechanism that validates pseudo-labels, ensuring high data quality without requiring time-consuming manual annotation for all samples. This self-service approach dramatically reduces annotation time while maintaining training data quality.

Inventive Principle:
Principle #25Self-service

Solution Approach 2:

The pipeline incorporates a filter mechanism that provides feedback on the quality of generated pseudo-labels. This feedback loop validates and refines the automatically generated masks, ensuring they meet quality thresholds before being used for training, thereby reducing the need for extensive manual verification and annotation time.

Inventive Principle:
Principle #23Feedback

Data Source

PatentUS20250239059A1Weakly-supervised referring expression segmentation
Publication Date: 2025.07.24 ADOBE INC
  • US20250239059A1 patent drawing
  • US20250239059A1 patent drawing
  • US20250239059A1 patent drawing

AI summary

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modifies parameters of a fused feature extractor. In particular, the disclosed systems generate inferred masks from digital images and digital text prompts using a fused feature extractor. Furthermore, the disclosed systems identify a subset of the inferred masks that satisfy a validity threshold. Moreover, the disclosed systems generate an augmented training set by combining the subset of the inferred masks with a training set that includes the ground truth masks. Further, the disclosed systems generate object mask predictions from the augmented training set and determine ground truth and pseudo measures of loss by comparing the object mask predictions with the inferred masks and the ground truth masks. From the ground truth and pseudo measures of loss, the disclosed systems modify parameters of the fused feature extractors.