Head-Mounted Display Viewport Reconstruction for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based streaming platforms face high latency issues due to network congestion and distance, leading to frustrating delays in real-time gaming experiences, as existing buffering methods can increase perceived latency when user input affects rendered content.
Innovation Solution
A head-mounted display device and method that predicts a user's viewpoint to reconstruct a viewport from a rendered rendering surface, using a processor to compute a predicted scene, determine actual user viewpoints, and interpolate pixel data to fill visual holes, thereby reducing perceived latency by pre-rendering content based on predicted user inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If content buffering is used to handle network latency, then playback continuity is improved, but user-perceived latency increases due to communication time lag between user input and content display
Solution Approach 1:
The system pre-renders content at multiple future timestamps based on predicted user navigation inputs before the actual inputs are received. This preliminary rendering action allows the content to be ready for immediate display when networked delays occur, eliminating the need to wait for user inputs to be processed and rendered in real-time, thus reducing user-perceived latency while maintaining playback continuity
2Adaptability or versatility
If cloud-based streaming is used to access vast content library, then content availability is improved, but network latency increases due to wide-area network distance
Solution Approach 1:
The system performs preliminary rendering of content at multiple future timestamps based on predicted user navigation inputs before actual user interactions occur. This advance preparation of content reduces the impact of wide-area network latency by ensuring content is already processed and ready for transmission, allowing users to access the vast content library with minimal perceived delay
3Manufacturing precision
If server-side rendering is used to provide high-end graphics on less powerful devices, then graphical performance is improved, but network bandwidth consumption increases
Solution Approach 1:
The system pre-renders content at multiple future timestamps based on predicted user navigation inputs. By predicting which content will be needed and rendering it in advance, the system reduces the total amount of data that needs to be transmitted over the network, as previously rendered content can be reused and combined with new renders rather than transmitting everything from scratch
Solution Approach 2:
The system dynamically adjusts rendering based on predicted user inputs and combines previously rendered content with new renders. This dynamic approach optimizes network bandwidth consumption by transmitting only the differences or updates rather than complete content sets, while maintaining high graphical performance through server-side rendering
Data Source
AI summary
A head mounted display device including a processor configured to compute a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the head mounted display device prior to the point in time, receive, from the user input device, a subsequent user navigation input near the point in time in the stream of user input, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, and reconstruct a viewport for the actual user viewpoint from the rendered rendering surface.