Occlusion Mask Prediction via Implicit Depth Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional augmented reality systems face challenges in accurately predicting occlusion masks, leading to poor integration of virtual objects with real-world environments, particularly around physical object edges, due to artifacts in depth estimation.
Innovation Solution
An occlusion mask prediction model comprising a feature network and a mask predictor is used to generate an occlusion mask from input images, considering the depth of virtual objects, which helps in seamlessly integrating virtual content with real-world images by determining occluded pixels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional depth map prediction is used to determine occlusion masks, then the process is simple and fast, but artifacts in depth estimation cascade down into the occlusion mask leading to poor augmentation quality around physical object edges
Solution Approach 1:
The system segments the occlusion mask generation process into two distinct components: a depth estimation network that predicts depth maps from input images, and an occlusion mask prediction model that uses both the depth maps and virtual object information to generate accurate occlusion masks. This segmentation allows each component to be optimized independently, with the mask prediction model specifically addressing edge artifact issues without requiring complete redesign of the entire system.
Solution Approach 2:
The depth estimation map serves as an intermediary between the input image and the final occlusion mask. Rather than directly generating occlusion masks from images, the system first produces depth estimates that capture geometric information, then uses these intermediates along with virtual object depth and position data to compute accurate occlusion relationships. This intermediary step prevents artifact propagation while maintaining computational efficiency.
2Manufacturing precision
If accurate depth estimation is performed to reduce artifacts, then occlusion mask quality improves, but processing time and computational resources increase
Solution Approach 1:
The depth estimation network performs preliminary depth prediction from input images before occlusion mask generation. By pre-computing depth information that captures the geometric structure of the environment, the system prepares essential data in advance that accelerates the subsequent occlusion mask prediction process. This preliminary action prevents the need for more computationally intensive methods during real-time mask generation.
Solution Approach 2:
The occlusion mask prediction model uses the depth estimation map as a key parameter, along with virtual object depth and position parameters, to efficiently compute occlusion relationships. By changing from direct image-based mask generation to parameter-based prediction using depth maps, the system achieves higher accuracy without proportional increases in processing time, as the depth parameters encapsulate essential geometric information in a compact form.
Data Source
AI summary
A system generates augmented reality content by generating an occlusion mask via implicit depth estimation. The system receives input image(s) of a real-world environment captured by a camera assembly. The system generates a feature map from the input image(s), wherein the feature map comprises abstract features representing depth of object(s) in the real-world environment. The system generates an occlusion mask from the feature map and a depth map for the virtual object. The depth map for the virtual object indicates a depth of each pixel of the virtual object. The occlusion mask indicates pixel(s) of the virtual object that are occluded by an object in the real-world environment. The system generates the composite image based on a first input image at a current timestamp, the virtual object, and the occlusion mask. The composite image may then displayed on an electronic display.