Sparse Depth Point Extraction for XR Rendering Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Lightweight extended reality (XR) systems, such as head-mounted displays with reduced processing power and limited power management, face challenges in maximizing performance while minimizing power consumption due to increased data throughput and compute load from rendering high-resolution depth maps.
Innovation Solution
The implementation of a method to generate sparse depth points from image and depth data, using multi-layer sampling and filtering techniques, which reduces data transmission and compute load by selectively extracting feature points where depth changes are significant, allowing for efficient reconstruction of depth maps on XR display devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution depth maps are transmitted to XR display devices, then image quality and rendering performance are improved, but network traffic and power consumption increase
Solution Approach 1:
The patent extracts only the essential depth information (sparse depth points at depth discontinuities) from the full depth map, transmitting only these extracted points to the XR display device rather than the complete high-resolution depth map, thereby reducing data transmission and power consumption while maintaining rendering quality
Solution Approach 2:
The depth map is segmented into significant regions (depth discontinuities) and non-significant regions (smooth depth areas). Only the significant regions containing depth discontinuities are transmitted as sparse depth points, while smooth regions are omitted, achieving data reduction without loss of critical depth information
2Measurement precision
If high-resolution depth maps are transmitted to XR display devices, then image quality and rendering performance are improved, but network traffic increases
Solution Approach 1:
The patent extracts only the essential depth information (sparse depth points at depth discontinuities) from the full depth map, transmitting only these extracted points to the XR display device rather than the complete high-resolution depth map, thereby reducing data transmission and power consumption while maintaining rendering quality
Solution Approach 2:
The depth map is segmented into significant regions (depth discontinuities) and non-significant regions (smooth depth areas). Only the significant regions containing depth discontinuities are transmitted as sparse depth points, while smooth regions are omitted, achieving data reduction without loss of critical depth information
3Measurement precision
If full-depth map data is processed on XR display devices, then rendering quality is maintained, but computational burden increases
Solution Approach 1:
The computing device extracts and transmits only the essential sparse depth points containing depth discontinuity information to the XR display device, eliminating the need for the XR device to process full-resolution depth maps and significantly reducing computational requirements while maintaining rendering quality
Solution Approach 2:
The computationally intensive depth map processing and feature extraction is performed in advance by the computing device before transmission to the XR display device. The XR device only receives and processes the pre-extracted sparse depth points, shifting the computational burden away from the XR device
Data Source
AI summary
A method includes accessing image data and depth data corresponding to image frames to be displayed on an extended reality (XR) display device, and determining sets of feature points corresponding to the image frames based on a multi-layer sampling of the image data and the depth data. The method further includes generating a set of sparse feature points based on an integration of the sets of feature points. The set of sparse feature points are determined based on relative changes in depth data with respect to the sets of feature points. The method further includes generating a set of sparse depth points based on the set of sparse feature points and the depth data and sending the set of sparse depth points to the XR display device for reconstruction of a dense depth map corresponding to the image frames utilizing the set of sparse depth points.