Mixed Reality Pass-Through Rendering with Asynchronous Depth Reprojection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional mixed-reality head-mounted display (HMD) pass-through systems suffer from loss of depth details, parallax errors, and computational latency when presenting virtual content, making it difficult for users to navigate and interact safely with their real-world environment.
Innovation Solution
The system processes images using processors to obtain texture and depth maps from stereo cameras, with asynchronous processing of depth maps to reduce latency, and late-stage reprojection to correct for user perspective changes, allowing for accurate and low-latency rendering of virtual representations of real-world environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If camera images are used to present real-world environment on HMD display, then pass-through functionality is provided to help users acclimate, but depth details of the real environment are lost
Solution Approach 1:
The patent introduces an intermediary optical system (beam splitter, mirrors, and optical waveguide) that allows camera images to pass through the HMD display while preserving depth information. This optical intermediary enables the user to see both the virtual display content and the real-world environment through the same device, maintaining depth perception without sacrificing pass-through functionality.
Solution Approach 2:
The patent transitions from a 2D camera image representation to a 3D optical path by introducing optical waveguides and beam splitters. This dimensional transformation allows the camera images to be projected at angles that preserve depth information while still passing through the display, enabling users to perceive real-world depth details through the HMD.
2Ease of operation
If camera modules are positioned to capture real-world environment, then pass-through view is provided, but parallax error occurs causing misalignment with user's view
Solution Approach 1:
The patent implements a feedback mechanism using head tracking sensors that continuously monitor the user's head position and orientation. This feedback information is used to dynamically adjust the optical path and image projection parameters, compensating for parallax errors and maintaining alignment between the camera view and the user's perspective in real-time.
Solution Approach 2:
The patent makes the optical system dynamic by introducing adjustable beam splitters and mirrors that can change their orientation based on real-time head tracking data. This dynamic adjustment allows the system to adapt to user movement and maintain proper image alignment, eliminating the fixed misalignment problem of conventional static optical systems.
3Measurement precision
If full-resolution depth map is calculated in real time, then accurate depth information is obtained, but computational expense increases and latency is introduced
Solution Approach 1:
The patent performs preliminary actions by pre-processing depth information and preparing optical correction parameters before the actual image display. Depth maps are calculated in advance and stored, allowing the system to quickly retrieve and apply the necessary depth data without performing expensive real-time calculations during the critical display moment, thus reducing latency while maintaining accuracy.
Solution Approach 2:
The patent segments the computational task by separating depth map generation from image processing. The depth map is calculated independently in advance, while the image processing focuses only on optical reprojection and parallax correction using the pre-computed depth data. This segmentation reduces the computational burden during real-time display while preserving depth information accuracy.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Systems and methods for providing a mixed-reality pass-through experience include implement acts of obtaining a texture map of a real-world environment, obtaining a depth map of the real-world environment, obtaining an updated texture map of the real-world environment subsequent to the obtaining of the depth map and the texture map, and rendering a virtual representation of the real-world environment utilizing both the depth map and the updated texture map that was obtained subsequent to the depth map. The texture map and the depth map may be based on a same image pair obtained from a pair of stereo cameras, the depth map being obtained by performing stereo matching on the same image pair. Additionally, the acts may further include detecting a predicted pose of a user and reprojecting a portion of the depth map to conform to a user perspective associated with the predicted pose.