Eye-Tracking Passthrough Rendering with Constant-Depth Meshes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in generating accurate depth representations for mixed-reality passthrough rendering due to computational complexity, power constraints, and environmental factors, leading to visual artifacts and motion sickness.
Innovation Solution
Utilizing eye-tracking information to determine the user's gaze direction and generate depth meshes with constant depth values for the user's eyes, prioritizing the region of interest, which are then used to reprojection images captured by cameras, thereby addressing the inaccuracies in the foreground and background.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by stationary object
If a continuous spatially varying depth mesh is used to approximate scene depth, then computational complexity is reduced and power consumption is lowered, but depth accuracy deteriorates in regions between foreground and background objects, causing visual artifacts
Solution Approach 1:
The patent segments the scene depth representation into multiple discrete depth planes instead of using a single continuous mesh. These depth planes are positioned at different depth values corresponding to foreground, midground, and background objects. By segmenting the depth space, the system maintains accurate depth representation for multiple objects without requiring a computationally expensive continuous mesh, thus resolving the contradiction between power consumption and depth accuracy.
Solution Approach 2:
The patent applies different depth representation strategies to different regions of the scene based on local requirements. Discrete depth planes are used where high accuracy is needed (at object locations), while peripheral regions use coarser approximation. This local differentiation allows the system to maintain depth accuracy where matters most while reducing computational complexity in less critical areas, balancing power consumption and accuracy.
2Measurement precision
If high-resolution and accurate depth sensing is implemented for the entire visible scene, then depth representation accuracy is improved, but computational complexity increases and processing time exceeds timing constraints
Solution Approach 1:
The patent implements partial depth sensing by focusing computational resources on capturing accurate depth information for specific regions of interest rather than the entire scene. By using discrete depth planes at key object locations and coarser sampling elsewhere, the system achieves sufficient depth accuracy for passthrough rendering while reducing the overall computational burden and meeting real-time processing constraints.
Solution Approach 2:
The patent changes the parameter of depth sampling resolution from uniform high-resolution across the entire scene to variable resolution with discrete depth planes at specific locations. This parameter transformation allows the system to maintain adequate depth accuracy where needed while dramatically reducing the total number of depth measurements required, thereby improving processing speed without sacrificing essential depth representation quality.
3Measurement precision
If discrete depth planes are used instead of continuous depth mesh, then depth accuracy is improved for multiple objects, but computational complexity increases
Solution Approach 1:
The patent segments the continuous depth space into aĉé number of discrete depth planes at strategically chosen depth values. This segmentation approach maintains depth accuracy for multiple foreground and background objects by ensuring each object is properly represented by at least one depth plane, while keeping the total number of planes manageable to avoid excessive computational complexity in the reprojection process.
Data Source
Figure 1
Figure 2A~2B
Figure 3A
AI summary
A method includes receiving scene image data, comprising left and right images for each eye. Through an eye tracking module, the method includes determining the user's gaze direction or eye vergence. Using this information, the method includes identifying an object in the scene that the user is focusing on, and determining, using a depth estimation module, a left depth from the left eye to the object and a right depth from the right eye to the object. Further, based on the computed left and right depths, the method includes generating, for the left and right eye, constant left and right depth meshes, generating a left and right output images by projecting the left and right images on the corresponding constant left and right depth meshes. Additionally, the method includes displaying the left output image for the left eye and displaying the right output image for the right eye.