User-Guided Object Segmentation for Accurate Media Removal

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for removing distracting objects from visual media items are arduous, incomplete, and often result in false positives or incomplete segmentation, affecting the user-perceived quality of images and videos.

Innovation Solution

A computer-implemented method that receives user input to identify objects for removal, translates it into a bounding box, uses a segmentation machine-learning model to generate a segmentation mask, and optionally inpaints the media item to remove the objects, with quality control through segmentation scores.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If automatic removal of distracting objects is performed, then object removal efficiency is improved, but false positives occur where additional objects are incorrectly removed

Engineering Contradiction:

Improveobject removal efficiencyVSAvoidaccuracy of object removal

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements feedback by allowing users to review and correct the segmentation masks generated by the AI model. Users can adjust the bounding boxes and segmentation results before final removal, ensuring that only the intended distracting objects are removed while preventing false positives of removing additional objects

Inventive Principle:

Principle #23Feedback

2Reliability

If manual image or video editing techniques are used to remove distracting objects, then accuracy of object removal is improved, but the process becomes arduous and time-consuming

Engineering Contradiction:

Improveaccuracy of object removalVSAvoidediting time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by automatically generating segmentation masks and identifying distracting objects before the user finalizes the removal. The AI model pre-processes the media item to create initial segmentation results, which the user can then review and adjust, significantly reducing the manual editing time while maintaining high accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary AI-based segmentation model that acts as a mediator between manual editing and automatic removal. This intermediary generates preliminary segmentation masks that guide the user's manual adjustments, combining the speed of automatic detection with the precision of manual control

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If automatic removal of distracting objects is performed, then productivity is improved, but incomplete segmentation results in portions of the removed object still being visible

Engineering Contradiction:

Improveobject removal efficiencyVSAvoidcompleteness of object removal

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system implements dynamics by allowing the segmentation mask to be dynamically adjusted by the user. The bounding boxes and segmentation boundaries are not fixed but can be modified interactively, enabling the user to refine the segmentation to ensure complete removal of the distracting object while maintaining high productivity through AI-assisted initialization

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12437358B2Performing segmentation of objects in media items based on user input

Publication Date: 2025.10.07 GOOGLE LLC

AI summary

A media application receives user input that indicates one or more objects to be erased from a media item. The media application translates the user input to a bounding box. The media application provides a crop of the media item based on the bounding box to a segmentation machine-learning model. The segmentation machine-learning model outputs a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask.