Amodal Instance Segmentation With Diffusion-Based Occlusion Completion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing machine learning models struggle to accurately identify and segment occluded regions of objects in images, particularly when the occlusion is large and complex, leading to difficulties in determining the shape, position, and size of hidden parts of objects.

Innovation Solution

A two-step framework using a mask network to predict instance and occlusion masks, followed by a diffusion model to complete the amodal segmentation mask, enabling the generation of complete images with occluded regions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If a mask network is used to predict instance masks, then the visible region of objects can be identified, but the occluded regions cannot be accurately determined

Engineering Contradiction:

Improvesegmentation accuracyVSAvoidoccluded region information

Core Design Contradiction:

Measurement precisionVSLoss of information

Solution Approach 1:

The method performs preliminary actions by first predicting the instance mask for visible regions, then using this prediction as input for the diffusion model to generate the complete amodal mask including occluded regions. This two-step preliminary action sequence enables accurate segmentation by building upon previous results.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The diffusion model serves as an intermediary between the instance mask prediction and the final amodal mask. It takes the instance mask as input and transforms it into a complete segmentation mask that includes both visible and occluded regions, effectively mediating the information flow and enabling recovery of hidden object parts.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Device complexity

If traditional segmentation methods are used, then the process is simple, but the ability to determine shape, position, and size of occluded objects is insufficient

Engineering Contradiction:

Improvemodel framework complexityVSAvoidoccluded object detection accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The method segments the segmentation task into two distinct parts: instance mask prediction for visible regions and diffusion-based completion for occluded regions. This segmentation of the overall task enables each component to specialize, with the diffusion model specifically handling the complex occluded region reconstruction.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The approach transitions from two-dimensional visible region detection to three-dimensional spatial reasoning by inferring occluded regions that extend beyond the visible boundaries. The diffusion model adds a temporal dimension through iterative generation, progressively building the complete object shape from the instance mask.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS12475565B2Amodal instance segmentation using diffusion models

Publication Date: 2025.11.18 ADOBE INC

AI summary

Systems and methods for instance segmentation are described. Embodiments include identifying an input image comprising an object that includes a visible region and an occluded region that is concealed in the input image. A mask network generates an instance mask for the input image that indicates the visible region of the object. A diffusion model then generates a segmentation mask for the input image based on the instance mask. The segmentation mask indicates a completed region of the object that includes the visible region and the occluded region.