Depth Reprojection with Adaptive Densification for VST XR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video see-through (VST) extended reality (XR) systems face challenges due to low-resolution and noisy sparse depth data, which affects the quality of rendered images, and existing methods for depth-based reprojection do not adequately address the need for high-resolution depth data to improve image rendering.
Innovation Solution
The technique involves predicting motion between image frames and performing depth densification and super-resolution to generate high-resolution depth data, using image and feature information to enhance depth data resolution, and reprojecting image frames for improved rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If depth-based reprojection is performed using existing low-resolution depth data, then the system can maintain simplicity and avoid complex processing, but the quality of rendered images deteriorates due to insufficient depth resolution
Solution Approach 1:
The system performs depth densification and super-resolution as preliminary processing steps before the actual reprojection operation. By pre-enhancing the depth data quality, the system prepares high-resolution depth maps that can be directly used in reprojection without requiring complex real-time processing during the rendering phase.
Solution Approach 2:
The patent introduces an intermediary processing pipeline that includes depth densification modules and super-resolution algorithms. These intermediaries transform the low-resolution depth data into high-resolution depth maps, serving as a bridge between the simple input depth data and the high-quality rendered output images.
2Manufacturing precision
If high-resolution depth data is generated through depth densification and super-resolution, then image rendering quality improves, but processing time increases causing higher latency
Solution Approach 1:
The depth densification and super-resolution operations are performed as preliminary actions before reprojection. By completing these computationally intensive tasks in advance or during low-priority processing windows, the system minimizes latency during the critical rendering and display phases.
Solution Approach 2:
The system maintains continuous processing of depth data through efficient pipeline design, where depth densification and super-resolution operations run continuously or in parallel with other system operations. This ensures that high-resolution depth data is always ready when needed without causing significant delays.
3Measurement precision
If motion prediction is performed between image frames, then reprojection accuracy improves by accounting for device movement, but processing complexity increases
Solution Approach 1:
Motion prediction is performed as a preliminary step before reprojection, where the system estimates device motion between captured image frames and uses this information to guide the reprojection process. This pre-computation of motion parameters simplifies the subsequent reprojection operation while maintaining high accuracy.
Solution Approach 2:
The system uses motion prediction results as feedback to adjust and optimize the reprojection process. By continuously monitoring and refining motion estimates, the system improves reprojection accuracy while managing processing complexity through adaptive algorithms that adjust to actual motion conditions.
Data Source
AI summary
A method includes obtaining a first image frame captured at a first time and first depth data associated with the first image frame, where the first image frame has a higher resolution than the first depth data. The method also includes predicting motion of the electronic device between the first time and a second time and generating second depth data based on the first depth data, the first image frame, and the predicted motion, where the second depth data has a higher resolution than the first depth data. The method further includes reprojecting the first image frame using the second depth data to generate a second image frame and displaying a rendered image based on the second image frame. Generating the second depth data includes performing depth densification and super-resolution in order to increase the resolution of the second depth data relative to the resolution of the first depth data.