Extended-Reality Texture Lookup Reduction Through Region-Level Depth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended-reality applications face significant performance and computational burdens due to the need for extensive texture lookups and comparisons when mixing virtual-reality (VR) and video-see-through (VST) content, especially in cases where the contents are intermixed, leading to increased processing demands and power consumption.
Innovation Solution
A method and system that divide the display area into regions, determine optical depths for VR and VST images, and fetch pixel data based on occlusion relationships, reducing the need for per-pixel depth comparisons and texture lookups by determining which content occludes the other within each region.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If per-pixel depth comparison is performed when VR and VST content are intermixed, then accurate occlusion rendering is achieved, but computational burden and processing time increase significantly
Solution Approach 1:
The patent divides the display area into multiple regions and performs depth comparison at the region level rather than per-pixel. For each region, it determines whether VR content or VST content occludes the other by comparing minimum and maximum optical depths, then applies the result to all pixels in that region. This segmentation approach maintains accurate occlusion rendering while dramatically reducing computational complexity from O(N pixels) to O(M regions) where M << N.
2Reliability
If per-pixel depth texture lookups are performed for intermixed VR and VST content, then correct depth comparison is achieved, but texture memory bandwidth consumption triples compared to full VR or full VST cases
Solution Approach 1:
The patent segments the display area into regions and performs depth comparison once per region by comparing minimum and maximum optical depths. This region-level comparison eliminates the need for triple texture lookups at every pixel, reducing texture memory bandwidth consumption from 3x to接近1x while maintaining depth comparison correctness through proper occlusion determination.
Solution Approach 2:
The patent performs preliminary depth analysis by determining minimum and maximum optical depths for each region before rendering. This preliminary action identifies occlusion relationships in advance, allowing the system to skip unnecessary per-pixel depth texture lookups and directly use the pre-determined occlusion results for all pixels in the region.
3Adaptability or versatility
If multiple VR images from different applications are rendered simultaneously, then diverse content is displayed, but the performance impact of depth lookups is multiplied
Solution Approach 1:
The patent extends the region-based approach to handle multiple VR images by performing depth comparisons at the region level for each VR application's content. This allows simultaneous rendering of multiple VR images with different applications while reducing the performance impact from O(N pixels per application) to O(M regions total), enabling the system to scale to multiple applications without linearly increasing the computational burden.
Data Source
AI summary
A display apparatus has a display area of an output image divided into a plurality of regions. For a given region of the output image, a corresponding first region in a video-see-through (VST) image and a corresponding second region in at least one virtual-reality (VR) image are determined. From a depth map corresponding to the VST image, a minimum optical depth and a maximum optical depth in the first region of the VST image are determined. From a depth map corresponding to the at least one VR image, a minimum optical depth and a maximum optical depth in the second region of the at least one VR image are determined. When the minimum optical depth in the first region of the VST image is greater than the maximum optical depth in the second region of the at least one VR image, pixel data for pixels in the given region of the output image from the second region of the at least one VR image is fetched. Otherwise, when the minimum optical depth in the second region of the at least one VR image is greater than the maximum optical depth in the first region of the VST image, the pixel data for the pixels in the given region of the output image from the first region of the VST image is fetched.