Shadow Map Reprojection for Low-Latency XR Image Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing extended reality (ER) systems suffer from latency and user discomfort due to delays in rendering and correcting images caused by conventional graphics processing units (GPUs), which render frames between 30 to 60 frames per second, leading to potential errors between the user's expected and displayed image positions.
Innovation Solution
Implementing a shadow map based late stage reprojection (LSR) process on GPUs, utilizing a shadow map mode that allows faster operation by discarding UV maps and recovering per pixel UV corrections from a Z buffer, enabling the GPU to operate at least twice as fast as traditional methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional GPUs are used for image rendering and late stage reprojection, then image quality and correction accuracy are maintained, but rendering speed is limited to 30-60 frames per second causing latency
Solution Approach 1:
The patent segments the LSR process into distinct computational stages: depth image processing to generate carrier geometry, forward projection of carrier geometry, rasterization to produce UV corrections, and final image resampling. This segmentation allows each stage to be optimized independently, with the GPU handling parallelizable operations at maximum speed while other components handle sequential processing.
Solution Approach 2:
The patent changes the computational parameters by switching from traditional full-resolution LSR operations to a simplified pipeline that uses downscaled depth images and carrier geometries. By processing at reduced resolution and then resampling the final output, the system achieves faster processing speeds while maintaining perceptual image quality.
2Measurement precision
If traditional LSR methods are used with full UV map processing, then accurate per-pixel corrections are achieved, but memory bandwidth consumption is high
Solution Approach 1:
The patent extracts only the essential correction information needed for LSR by discarding the full UV map after generating per-pixel UV corrections during rasterization. Instead of storing and processing complete UV maps, the system extracts only the correction data required for the final resampling operation, significantly reducing memory bandwidth requirements.
Solution Approach 2:
The patent discards intermediate UV map data that would otherwise consume significant memory bandwidth, then recovers the necessary per-pixel correction information directly from the rasterization output and depth buffer. This approach maintains correction accuracy while minimizing memory usage by discarding redundant data structures.
Data Source
AI summary
Techniques for improving how LSR is performed are disclosed. A service accesses a depth image, a pose correction matrix, and a color image. The service extracts a carrier geometry from the depth image. The service forward projects the LSR carrier geometry by multiplying each vertex of the LSR carrier geometry with the pose correction matrix. While the GPU is operating in a shadow map mode, the service causes the GPU to perform a rasterization process to produce a UV map and a Z buffer. The service discards the UV map, resulting in the per pixel UV corrections included in the UV map also being discarded. The service recovers the per pixel UV corrections using the Z buffer. The service uses the recovered per pixel UV corrections to resample the color image, resulting in generation of a corrected color image.