Media Object Removal via Bounding Box Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for removing distracting objects from visual media items are arduous, incomplete, and prone to false positives or incomplete segmentation, leading to unsatisfactory user-perceived quality.
Innovation Solution
A computer-implemented method that utilizes user input to generate a bounding box, applies a segmentation machine-learning model to create a segmentation mask, and uses an inpainting model to remove the identified objects, with optional validation and correction mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual image or video editing techniques are used to remove distracting objects, then users can remove objects, but the process becomes arduous and incomplete
Solution Approach 1:
The patent replaces manual mechanical editing operations with an automated machine-learning-based segmentation system. The segmentation model automatically identifies and segments distracting objects based on user input, eliminating the need for manual pixel-by-pixel editing while ensuring complete object removal through algorithmic precision.
Solution Approach 2:
The segmentation model performs self-service by automatically identifying and segmenting objects without requiring continuous user intervention. Once the user provides initial input (such as selecting a distracting object), the model autonomously completes the segmentation process, generating masks and guiding the removal operation without further manual effort.
2Extent of automation
If automatic removal of distracting objects is implemented, then the process becomes automated, but false positives occur where additional objects or portions of objects are removed
Solution Approach 1:
The patent implements feedback mechanisms where the segmentation model's output is evaluated and refined. The system provides feedback loops that allow users to correct segmentation errors, and the model learns from these corrections to improve future segmentation accuracy, reducing false positives while maintaining high automation levels.
Solution Approach 2:
The segmentation model initially performs partial segmentation, identifying only the most certain portions of distracting objects. Users can then guide the model to expand or refine the segmentation as needed, allowing the system to start with conservative, high-precision segmentation and progressively include more areas based on user feedback and confidence thresholds.
3Extent of automation
If automatic removal of distracting objects is implemented, then the process becomes automated, but incomplete segmentation results in portions of the removed object still being visible
Solution Approach 1:
The patent performs preliminary segmentation actions that identify and isolate distracting objects before the actual removal process. The segmentation model pre-processes the image to create accurate masks of objects to be removed, ensuring complete coverage of distracting elements before the inpainting or removal operation begins, preventing any residual visibility.
Solution Approach 2:
The patent applies segmentation to divide the image into distinct regions: distracting objects to be removed and preserved content. This segmentation creates precise boundaries and masks that guide the removal process, ensuring that entire objects are removed while preserving surrounding areas, thereby achieving complete object removal without leaving visible remnants.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A media application receives user input that indicates one or more objects to be erased from a media item. The media application translates the user input to a bounding box. The media application provides a crop of the media item based on the bounding box to a segmentation machine-learning model. The segmentation machine-learning model outputs a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask.