Image-guided depth propagation for space-warping rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In extended reality (XR) environments, latency issues persist due to the need for concurrent transmission of color and depth information, causing image misalignment with the user's head pose, especially as image sizes and resolution increase, despite advancements in high-speed wireless protocols.
Innovation Solution
The implementation of image-guided depth propagation using sparse depth points extracted from a source depth map, which allows for space-warping of images without transmitting the entire depth map, reducing data transmission load and enabling accurate rendering of images aligned with the user's head pose.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the entire depth map is transmitted to the display device, then accurate space-warping rendering can be achieved, but data transmission load and latency increase
Solution Approach 1:
The patent extracts only the essential depth information (sparse depth points at feature locations) from the complete depth map and transmits only these extracted points to the display device. This selective extraction maintains sufficient rendering accuracy while dramatically reducing the data transmission volume and associated latency.
Solution Approach 2:
The depth map is segmented into sparse depth points located at feature positions rather than transmitting the complete dense depth map. This segmentation approach preserves the critical depth information needed for accurate rendering while minimizing transmission data requirements.
2Manufacturing precision
If high-resolution images are transmitted, then image quality improves, but transmission time and latency increase
Solution Approach 1:
The patent extracts only the necessary depth information (sparse points at features) rather than transmitting the complete high-resolution depth map. This selective extraction maintains image quality for space-warping rendering while significantly reducing transmission data volume and improving transmission speed.
3Measurement precision
If depth information is transmitted for every pixel, then accurate depth mapping is achieved, but data transmission load increases
Solution Approach 1:
The patent extracts only depth information at feature points (sparse depth points) rather than transmitting depth for every pixel. This selective extraction maintains sufficient depth mapping accuracy for rendering while dramatically reducing the quantity of data that must be transmitted.
Solution Approach 2:
The depth data is segmented into discrete sparse points at feature locations rather than a continuous dense map. This segmentation preserves the essential depth information needed for accurate rendering while minimizing the total data volume.
Data Source
AI summary
Updating an image during real-time rendering of images by a display device can include determining a depth for each pixel of a color frame received from a source device and corresponding to the image. Each pixel's depth is determined by image-guided propagation of depths of sparse points extracted from a depth map generated at the source device. With respect to pixels corresponding to an extracted sparse depth point, image-guided depth propagation can include retaining the depth of the corresponding sparse depth point unchanged from the source depth map. With respect to each pixel corresponding to a non-sparse depth point, image-guided depth propagation can include propagating to the corresponding non-sparse depth point a depth of a sparse depth point lying within a neighborhood of the non-sparse depth point. Pixel coordinates of the color frame can be transformed for generating a space-warped rendering of the image.