Amodal Segmentation via Conditional Diffusion Models

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing computer vision technologies struggle with zero-shot amodal segmentation, where they cannot generate complete, whole representations of occluded objects in images without prior training on specific datasets.

Innovation Solution

The use of a conditional generative model, specifically a conditional diffusion model, trained on a synthetically curated dataset of occluded and whole object pairs, allows for the generation of amodal images of occluded objects in input images.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional computer vision methods are used for amodal segmentation, then the system can process images with standard algorithms, but it cannot generate complete representations of occluded objects without prior training on specific datasets

Engineering Contradiction:

Improvezero-shot amodal segmentation capabilityVSAvoidaccuracy of object completion

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system performs preliminary actions by pre-training the generative model on a large dataset of whole objects before deployment. This pre-training enables the model to learn general object representations and patterns, which are then applied during zero-shot amodal segmentation to accurately complete occluded objects without requiring task-specific training data

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses a generative model to create synthetic copies of whole objects that can be overlaid onto occluded regions in input images. By generating realistic object completions through the conditional diffusion model, the system produces accurate amodal segmentations that mimic what the complete object would look like without occlusion

Inventive Principle:

Principle #26Copying

2Adaptability or versatility

If a conditional diffusion model is trained on synthetically curated data, then zero-shot amodal segmentation performance improves, but the training process becomes more complex

Engineering Contradiction:

Improvegeneralizability across diverse scenariosVSAvoidtraining dataset curation process

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system employs automated procedures to generate synthetic training data by rendering 3D models of objects in various occluded configurations. This self-service approach to data generation eliminates the need for manual annotation of occluded objects, allowing the model to learn from large volumes of synthetically generated examples with diverse occlusion patterns and viewpoints

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The synthetically curated training dataset serves multiple functions: it provides training examples for the conditional diffusion model, enables zero-shot generalization to unseen object categories, and supports various amodal segmentation scenarios. The same synthetic data infrastructure supports both training and evaluation phases across different application domains

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20250182358A1Amodal segmentation by synthesizing whole objects

Publication Date: 2025.06.05 TOYOTA RESEARCH INSTITUTE INC

AI summary

Systems and methods are provided for generating amodal images from occlusions objects in input images. Examples herein include receiving a prompt selecting an object in an input image; applying the input image to a trained conditional generative model that generates an amodal image of the selected object based on the prompt and the input image; and outputting the amodal image.