AR Occlusion Handling Using Combined 3DRU and Stereo Depth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graphics processing systems struggle with occlusion handling between real and virtual objects in augmented reality, as 3DRU meshes are not immediately available, and depth maps from stereo cameras are noisy and not available at display refresh rates, leading to poor occlusion quality.
Innovation Solution
Combining 3DRU depth and depth from stereo to create a reliable depth map that captures both static and dynamic objects, using a re-projection module, depth combiner, and compositor to achieve seamless occlusion handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth maps from stereo cameras are used for occlusion handling, then dynamic objects can be captured, but the depth maps are noisy and not available at display refresh rates
Solution Approach 1:
The patent combines 3DRU depth maps (which are clean but lack dynamic objects) with depth maps from stereo cameras (which capture dynamic objects but are noisy) to create a composite depth map that achieves both reliability and temporal coverage. The depth combiner module merges these two sources using confidence maps and hole masks to produce a reliable depth representation at display refresh rates.
2Reliability
If 3DRU meshes are used for depth estimation, then clean depth maps are obtained, but the meshes are not immediately available and miss dynamic objects
Solution Approach 1:
The system performs preliminary action by using 3DRU depth maps as a baseline or fallback source that is continuously available. While 3DRU meshes are being constructed, the system already has clean depth data from 3DRU that can be combined with stereo depth data, ensuring that depth estimation never stalls while waiting for 3DRU mesh completion.
Solution Approach 2:
The patent merges 3DRU depth maps with stereo depth maps to compensate for the limitations of each individual source. The 3DRU provides clean baseline depth while stereo provides dynamic object coverage, and their combination resolves both the availability timing issue and the dynamic object detection issue.
3Reliability
If separate processing is performed for 3DRU depth and stereo depth, then each depth source can be optimized independently, but the system complexity increases
Solution Approach 1:
The patent merges the separate 3DRU depth processing pipeline with the stereo depth processing pipeline through a depth combiner module. This allows independent optimization of each depth source while providing a unified interface that produces a single composite depth map, balancing independent optimization with system integration.
Solution Approach 2:
The depth combiner module serves multiple functions: it combines 3DRU and stereo depth maps, generates confidence maps, creates hole masks, and produces the final composite depth map. This multi-functional approach reduces overall system complexity by consolidating multiple processing tasks into a single universal module.
Data Source
AI summary
The present disclosure relates to methods and apparatus for graphics processing. The apparatus can determine a combined depth map based on at least one of a 3DRU mesh, a DFS map, and a display pose. The apparatus can also determine one or more re-projected eye and depth buffers based on at least one of one or more eye buffers, one or more depth buffers, one or more bounding box and warp parameters, and the display pose. Further, the apparatus can communicate at least one composited frame based on the combined depth map and the one or more re-projected eye and depth buffers. The apparatus can also determine the at least one composited frame based on the combined depth map and the one or more re-projected eye and depth buffers. The apparatus can also combine the 3DRU mesh, the DFS map, and the display pose.