Mixed Reality Timewarp for Cloud Rendering Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed reality (MR) systems face significant latency issues due to computationally intensive algorithms and asynchronous processing of virtual and real-world elements, leading to outdated scene rendering and motion sickness in applications like flight simulators.
Innovation Solution
The implementation of a timewarp technique that predicts head poses and applies corrections to both virtual and stereoscopic imagery, synchronizing the two to minimize latency and ensure accurate representation of the user's perspective, even with asynchronous rendering and camera shutter variations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If computationally intensive algorithms are used for high frame rate and high resolution rendering, then rendering quality is improved, but rendering latency increases
Solution Approach 1:
The system performs preliminary actions by predicting the head pose at the time of display before the actual rendering is complete. This prediction allows the system to pre-calculate the necessary warping parameters, so that when the rendered image becomes available, it can be quickly adjusted to match the predicted head position, reducing the perceived latency without compromising rendering quality
Solution Approach 2:
The patent introduces an intermediary warping layer between the rendered image and the display. This warping mechanism acts as a mediator that adjusts the rendered image based on predicted head pose, allowing the system to decouple the rendering process from the display timing and reduce latency without sacrificing image quality
2Productivity
If heavy pipelining is applied to maintain throughput, then processing efficiency is improved, but scene rendering becomes outdated
Solution Approach 1:
The system dynamically adjusts the rendering pipeline by allowing multiple frames to be processed simultaneously at different stages. While the pipeline maintains high throughput, the warping step is dynamically updated using the latest head pose predictions, ensuring that even though processing is pipelined, the final displayed image reflects the most current head position
3Device complexity
If cloud rendering is used for flight simulators, then computational load is reduced, but system latency increases
Solution Approach 1:
The client device performs preliminary actions by predicting head pose and calculating warping parameters locally before receiving the rendered image from the cloud. This allows the system to compensate for network latency through predictive warping, reducing the perceived system latency while maintaining the computational advantages of cloud rendering
Solution Approach 2:
The system implements a feedback mechanism where the predicted head pose information is continuously updated and used to adjust the warping parameters. This feedback loop allows the client to compensate for cloud rendering delays by continuously adjusting the displayed image based on the latest head position predictions
4Productivity
If asynchronous processing of virtual and real-world elements is implemented, then processing efficiency is improved, but synchronization accuracy deteriorates
Solution Approach 1:
The patent introduces a warping intermediary that synchronizes the asynchronous virtual and real-world elements. By applying head-pose-based warping to both the virtual reality imagery and the stereoscopic camera imagery, the system creates a common reference frame that aligns the two asynchronous streams, maintaining synchronization accuracy while preserving processing efficiency
Data Source
AI summary
A mixed reality (MR) system is disclosed. The MR system may determine a first predicted head pose corresponding to a time that virtual reality imagery is rendered, determine a second predicted head pose corresponding to a selected point in time during a camera shutter period, and combine the virtual reality imagery with the stereoscopic imagery based on the first predicted head pose and the second predicted head pose. A simulator that employs remote (e.g., cloud) rendering is also disclosed. The simulator/client device may determine a first pose (e.g., vehicle pose and/or head pose), receive video imagery rendered by a remote server based on the first pose, and apply a timewarp correction to the video imagery based on a comparison of the first pose and a second pose.