Continuous Time Warp for Low-Latency VR/AR Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing VR and AR systems face challenges in providing a comfortable, natural-feeling experience due to the complexity of human visual perception and nervous system, particularly in modifying virtual images in real-time to account for user movement, leading to undesired experiences from pose mismatch and image tearing.
Innovation Solution
The implementation of continuous time warp and binocular time warp methods to transform images based on updated user positions, reducing motion-to-photon latency and ensuring seamless integration of virtual content with the real world, using techniques like read cursor redirection, pixel redirection, and asynchronous warping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system generates and displays content at a fixed refresh rate (e.g., 60 Hz), then the system maintains stable operation, but the motion-to-photon latency increases causing pose mismatch and degraded user experience
Solution Approach 1:
The system dynamically adjusts the display timing of individual pixels based on the user's head pose changes. Instead of a fixed refresh rate for all pixels, the display device scans and updates pixels at different times, with earlier pixels in the frame being displayed sooner and later pixels being warped and displayed later to compensate for head movement. This dynamic timing adjustment reduces motion-to-photon latency while maintaining system stability.
Solution Approach 2:
The system performs time warp operations on the image frame before it is fully scanned and displayed. By predicting the user's head pose and pre-warping the image frame based on this prediction, the system compensates for anticipated head movements before they affect the displayed content, thereby reducing latency and improving the motion-to-photon response time.
2Measurement precision
If the system applies post-processing correction for user pose changes on buffered images, then the user pose mismatch is corrected, but image tearing occurs on scanning displays
Solution Approach 1:
The system segments the image frame into multiple regions corresponding to different scan lines or pixel groups. Each segment is warped independently based on its position in the scan sequence and the predicted head pose at the time it will be displayed. This segmentation allows the system to apply pose correction to each pixel group at the appropriate time without causing temporal inconsistencies that lead to image tearing.
Solution Approach 2:
The system dynamically adjusts the warping parameters for different regions of the image frame based on the scan sequence and predicted head pose timing. Earlier-scanned pixels use different warping parameters than later-scanned pixels, creating a dynamic, time-varying transformation that maintains spatial accuracy throughout the entire frame display without causing tearing artifacts.
3Ease of operation
If the system uses conventional time warp methods, then the user experience is improved, but spatial registration of virtual objects deteriorates with scanning displays
Solution Approach 1:
The system implements a dynamic time warp approach where the warping transformation is continuously adjusted based on the scan sequence and predicted head pose. Each pixel or pixel group is warped at the specific moment it is scanned and displayed, using the predicted head pose for that particular time instant. This dynamic adjustment maintains accurate spatial registration of virtual objects with the real world throughout the entire frame display, overcoming the limitations of conventional static time warp methods on scanning displays.
Solution Approach 2:
The system performs preliminary warping of the image frame based on predicted head pose before the scanning display begins its scan. This pre-warping ensures that when pixels are scanned and displayed in sequence, they are already positioned correctly relative to the predicted head pose at display time, maintaining spatial registration accuracy without requiring complex real-time adjustments during the scan.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Embodiments of the present disclosure relate to continuous and/or binocular time warping methods to account for head movement of the user without having to re-render a displayed image. Continuous time warping allows for transformation of an image from a first perspective to a second perspective of the viewer without having to re-render the image from the second perspective. Binocular time warp refers to the late-frame time warp used in connection with a display device including a left display unit for the left eye and a right display unit for the right eye where the late-frame time warp is performed separately for the left display unit and the right display unit. Warped images are sent to the left and the right display units where photons are generated and emitted toward respective eyes of the viewer, thereby displaying an image on the left and the right display units at the same time.