VST XR Reprojection Using Foreground Depth and Constant-Depth Backgrounds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video see-through (VST) XR systems face challenges such as high computational resource demands and latency due to depth-based reprojection, particularly at high resolutions, which can cause noticeable delays and limit their adoption.
Innovation Solution
Implement depth-varying reprojection techniques that differentiate between foreground and background objects, using depth-based reprojection for foreground objects and constant-depth reprojection for background objects, reducing computational load and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth-based reprojection is performed for all objects in the scene, then accurate depth reconstruction is achieved, but computational resource demands and latency increase significantly
Solution Approach 1:
The patent segments the scene into foreground objects and background objects based on depth information. Depth-based reprojection is applied selectively to foreground objects where accuracy is critical, while background objects use constant-depth reprojection. This segmentation resolves the contradiction by applying different processing strategies to different parts of the scene, maintaining accuracy where needed while improving overall computational efficiency.
Solution Approach 2:
The patent applies local quality by using depth-based reprojection (higher quality) for foreground objects and constant-depth reprojection (lower quality) for background objects. This localized application of different reprojection methods ensures that computational resources are concentrated on areas requiring high accuracy while reducing processing demands in less critical areas, thereby resolving the trade-off between accuracy and efficiency.
2Manufacturing precision
If depth-based reprojection is performed at high resolutions, then image quality is improved, but computation time increases noticeably
Solution Approach 1:
The patent applies partial action by performing computationally intensive depth-based reprojection only on foreground objects rather than all objects in the scene. This partial application of the complex algorithm reduces overall computation time while maintaining high image quality for the most important visual elements, effectively resolving the contradiction between image quality and computation time.
3Productivity
If constant-depth reprojection is used for background objects, then computational load is reduced, but hole artifacts may increase
Solution Approach 1:
By segmenting the scene into foreground and background regions, the patent can apply constant-depth reprojection to background objects where hole artifacts are less noticeable, while maintaining depth-based reprojection for foreground objects. This segmentation allows the system to tolerate some artifacts in less critical areas while preserving quality where it matters most, resolving the contradiction between computational efficiency and artifact generation.
Data Source
AI summary
A method includes obtaining images of a scene captured using a stereo pair of imaging sensors of an XR device and depth data associated with the images, where the scene includes multiple objects. The method also includes obtaining volume-based 3D models of the objects. The method further includes, for one or more first objects, performing depth-based reprojection of the one or more 3D models of the one or more first objects to left and right virtual views based on one or more depths of the one or more first objects. The method also includes, for one or more second objects, performing constant-depth reprojection of the one or more 3D models of the one or more second objects to the left and right virtual views based on a specified depth. In addition, the method includes rendering the left and right virtual views for presentation by the XR device.