Passthrough Occlusion Rendering via Segmented Depth Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial reality systems with head-mounted displays (HMDs) struggle to accurately render occlusions in passthrough visualizations, leading to incorrect boundaries between virtual and physical objects due to imprecise 3D model representations.
Innovation Solution
The system generates instance segmentation masks for objects in the captured image and computes depth measurements based on corresponding 3D model portions, assigning significantly farther depth values to the outer edges of the masks to enhance depth contrast, thereby improving occlusion rendering in passthrough visualizations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a 3D model representation is used for passthrough re-rendering, then parallax and misalignment compensation are improved, but occlusion boundary precision deteriorates
Solution Approach 1:
The patent segments the passthrough image by generating instance segmentation masks that identify and separate different objects (e.g., persons, pets) from the background. This segmentation allows independent depth computation and occlusion handling for each object, resolving the boundary precision issues that arise when using a single 3D model for the entire scene.
Solution Approach 2:
The patent applies local quality by computing depth measurements specifically for the segmented object regions rather than for the entire image. By assigning depth values locally to each segmented object based on its corresponding 3D model portion, the system achieves higher occlusion boundary precision where it matters most while maintaining the benefits of 3D re-rendering.
2Measurement precision
If depth measurements are computed from 3D model portions, then occlusion rendering accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary action by generating instance segmentation masks and identifying corresponding 3D model portions before the actual depth computation and rendering process. This pre-processing step organizes the data structure so that depth measurements can be computed efficiently for each segmented object rather than processing the entire scene at once, reducing overall computational complexity.
Solution Approach 2:
The patent extracts only the necessary depth information from the segmented object regions and their corresponding 3D model portions, rather than computing and processing all possible depth data. This extraction approach focuses computational resources on the critical occlusion boundaries while ignoring redundant information, thereby reducing computational complexity.
Data Source
AI summary
In one embodiment for generating passthrough, a system may receive an image and depth measurements of an environment and generate a corresponding 3D model. The system identifies, in the image, first pixels depicting a physical object and second pixels corresponding to a padded boundary around the first pixels. The system associates the first pixels with a first portion of the 3D model representing the physical object and a first representative depth value computed based on the depth measurements. The system associates the second pixels with a second portion of the 3D model representing a region around the physical object and a second representative depth value farther than the first representative depth value. The system renders an output image depicting a virtual object and the physical object. Occlusions between the virtual object and the physical object are determined using the first representative depth value and the second representative depth value.