User-Guided Object Segmentation for Accurate Media Erasing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for removing distracting objects from visual media items are arduous, incomplete, and prone to false positives or incomplete segmentation, affecting the user-perceived quality of images and videos.
Innovation Solution
A computer-implemented method that receives user input to identify objects for removal, translates it into a bounding box, uses a segmentation machine-learning model to generate a segmentation mask, and optionally employs an inpainting model to erase the objects, with quality control through segmentation scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automatic object removal is performed, then productivity is improved, but reliability deteriorates due to false positives and incomplete segmentation
Solution Approach 1:
The system implements feedback by presenting segmentation results to users for verification and correction. Users can review the automatically segmented objects and provide feedback on incorrect segmentations, which is then used to refine the model's future predictions, thereby improving reliability while maintaining automation.
Solution Approach 2:
The system performs partial automation by automatically removing only the confidently segmented objects while leaving ambiguous cases for manual review. This approach balances productivity improvement with reliability maintenance by not forcing complete automation on all cases.
2Reliability
If manual image or video editing techniques are used to remove distracting objects, then reliability is improved, but productivity deteriorates due to the arduous and time-consuming nature of the task
Solution Approach 1:
The system performs preliminary automated segmentation and object identification before final removal. This preliminary action handles the bulk of the work automatically, improving productivity, while users only need to perform final verification and correction, maintaining reliability without requiring complete manual editing.
3Productivity
If complete automation is implemented for object removal, then productivity is improved, but device complexity increases due to the need for advanced machine learning models and processing infrastructure
Solution Approach 1:
The system uses an intermediary human reviewer as a mediator between the automated segmentation model and the final output. This intermediary layer allows the use of relatively simple automated models while maintaining high reliability through human oversight, balancing productivity improvement with manageable system complexity.
Data Source
AI summary
A media application receives user input that indicates one or more objects to be erased from a media item. The media application translates the user input to a bounding box. The media application provides a crop of the media item based on the bounding box to a segmentation machine-learning model. The segmentation machine-learning model outputs a segmentation mask for one or more segmented objects in the crop of the media item and a corresponding segmentation score that indicates a quality of the segmentation mask.