Depth Map Re-projection for XR Network Interruption Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended reality (XR) systems experience abrupt termination of user experiences due to reliability issues with wireless network connections, leading to a poor user experience when image data interruptions occur.
Innovation Solution
Implementing a method to re-project depth maps on user electronic devices by accessing and storing feature points from a depth map, allowing for partial re-rendering of objects based on spatiotemporal data, head pose, and object pose information, even when the wireless network connection is unavailable, thereby maintaining a graceful termination of the XR experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If wireless network connection is used to transmit image data, then device portability and flexibility are improved, but connection reliability deteriorates causing abrupt termination of XR experience
Solution Approach 1:
The system performs preliminary actions by storing depth map feature points and performing re-projection calculations in advance, before network disconnection occurs. This allows the XR device to continue rendering using locally stored data when the wireless connection fails, preventing abrupt termination of the user experience.
2Manufacturing precision
If full image data is transmitted from computing platform to XR device, then rendering quality is improved, but network bandwidth consumption and data loss risk increase
Solution Approach 1:
The system extracts only the essential feature points from depth maps rather than transmitting complete image data. These feature points contain the critical spatial and temporal information needed for re-projection, significantly reducing data transmission requirements while maintaining adequate rendering quality for graceful degradation during network interruptions.
Solution Approach 2:
The system creates simplified copies of the scene using feature points from depth maps, which can be re-projected on the XR device without requiring the original full-resolution image data. This copying approach reduces network dependency and data loss impact.
3Reliability
If depth map re-projection is implemented using feature points, then network interruption tolerance is improved, but computational complexity on XR device increases
Solution Approach 1:
The system performs partial re-projection using only the essential feature points rather than processing complete image data. This partial action approach provides sufficient visual continuity during network interruptions while keeping computational requirements manageable for the XR device.
Data Source
AI summary
A method includes rendering, on displays of an extended reality (XR) display device, a first sequence of image frames based on image data received from an external electronic device associated with the XR display device. The method further includes detecting an interruption to the image data received from the external electronic device, and accessing a plurality of feature points from a depth map corresponding to the first sequence of image frames. The plurality of feature points includes movement and position information of one or more objects within the first sequence of image frames. The method further includes performing a re-warping to at least partially re-render the one or more objects based at least in part on the plurality of feature points and spatiotemporal data, and rendering a second sequence of image frames corresponding to the partial re-rendering of the one or more objects.