Predictive Overfilling for VR Latency Compensation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtual reality (VR) services, especially remote ones, latency issues due to processing and network delays cause inconsistencies between the generated VR video frame and the user's field of view, leading to motion sickness and an uncomfortable user experience.
Innovation Solution
A method involving head tracking position information to predict a user's future head position, determining an overfill factor based on expected error, and rendering an overfilled image to account for latency, with dynamic adjustments in pixel dimensions or aspect ratio according to connection latency and head motion, ensuring timely display of VR content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If VR content is processed remotely at a server, then computing power requirements are reduced at the client device, but latency increases due to encoding, decoding, and network transfer
Solution Approach 1:
The system performs preliminary head position prediction before the actual display time. By predicting where the user's head will be at the scan-out time and pre-calculating the overfill factor based on expected error margins, the system prepares the VR content in advance to account for upcoming latency, ensuring the displayed content aligns with the user's predicted field of view rather than their current position
2Measurement precision
If the VR video frame is rendered to match the current field of view, then accuracy is improved, but motion sickness increases due to inconsistency with the user's actual field of view at display time
Solution Approach 1:
The system predicts the user's future head position at the scan-out time and renders the VR content accordingly. By using predicted head position rather than current position, the system proactively adjusts the field of view to match where the user will be looking, eliminating the inconsistency that causes motion sickness
Solution Approach 2:
The system applies an overfill factor that creates a buffer zone around the predicted field of view. This cushioning accounts for potential prediction errors and ensures that even if the predicted head position is slightly off, the displayed content will still align with the user's actual field of view, preventing motion sickness
3Device complexity
If a fixed rendering size is used, then processing is simplified, but adaptability to changing latency and head motion is reduced
Solution Approach 1:
The system dynamically adjusts the rendering size and overfill factor based on real-time conditions. The overfill factor changes according to network latency variations and head motion characteristics, allowing the system to adapt to changing conditions while maintaining computational efficiency through targeted adjustments rather than complete reconfiguration
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
An example disclosed method in accordance with some embodiments includes: receiving head tracking position information from a client device, the head tracking position information associated with a user at the client device; predicting a future head position of the user at a scan-out time for displaying a virtual reality (VR) video frame, wherein the VR video frame is displayed to the user via the client device; determining an overfill factor based on an expected error in the predicted future head position of the user; rendering an overfilled image based on the predicted future head position of the user and the overfill factor; and sending the VR video frame including the overfilled image to the client device for display to the user.