Mixed Reality Object Removal via 3D Model Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional real-time or near real-time Augmented Reality (AR) and Mixed Reality (MR) methods often suffer from inaccuracies and artifacts due to the use of inpainting techniques, which degrade the quality of rendered images and detract from the user experience.
Innovation Solution
The method involves obtaining live images with a tracked object, determining the camera pose, rendering synthetic images without the object from a 3D model, computing optical flow magnitudes, and determining replacement pixels based on corresponding pixels in synthetic images to enhance image processing and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If inpainting techniques are used to remove tracked objects from live images, then real-time processing is achieved, but image quality deteriorates due to artifacts and inaccuracies
Solution Approach 1:
The system performs preliminary 3D reconstruction of the scene using multiple images captured at different times, creating a pre-computed 3D model that can be queried to generate synthetic background images. This preliminary action enables high-quality rendering without requiring complex real-time inpainting operations.
Solution Approach 2:
Instead of modifying the original live image through inpainting, the system copies information from pre-computed 3D model data to generate synthetic images that replace the tracked object. This copying approach from 3D model data eliminates the artifacts inherent in pixel-based inpainting while maintaining real-time performance.
2Device complexity
If conventional inpainting methods are used for object removal, then processing simplicity is maintained, but reconstruction accuracy decreases due to lack of 3D spatial understanding
Solution Approach 1:
The system transitions from 2D image-based inpainting to 3D space-based reconstruction. By capturing multiple images at different times and positions, the system builds a 3D understanding of the scene, enabling accurate reconstruction that accounts for depth, occlusion, and spatial relationships—dimensions lost in conventional 2D inpainting.
Solution Approach 2:
The system performs preliminary 3D reconstruction using multiple historical images before processing the current live image. This pre-computed 3D model provides accurate spatial information that guides the object removal process, achieving high reconstruction accuracy without complicating the actual time-critical processing.
3Speed
If real-time camera tracking is performed, then responsiveness is improved, but tracking accuracy decreases due to accumulated errors in live processing
Solution Approach 1:
The system performs camera pose estimation and 3D reconstruction using a series of historical images captured before the current frame. This preliminary reconstruction establishes an accurate reference frame that reduces accumulated tracking errors, while still maintaining real-time responsiveness by processing only the current frame against the pre-computed model.
Data Source
AI summary
Disclosed embodiments pertain to apparatus, systems, and methods for mixed reality. In some embodiments, a camera pose relative to a tracked object in a live image may be determined and used to render synthetic images from keyframes in a 3D model without the tracked object. Optical flow magnitudes for pixels in a first mask region relative to a subset of the synthetic images may be determined and the optical flow magnitudes may be used to determine pixels in each of the subset of synthetic images that correspond to pixels in the first mask. For each pixel in the first mask, a corresponding replacement pixel may be determined as a function of pixels in the subset of synthetic images that correspond to the corresponding pixel in the first mask.