Parallax Occlusion Rendering for VR Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Interactive video streaming systems face significant latency issues, particularly in virtual reality games, due to poor correspondence between the user's current camera perspective and the displayed frames, leading to poor user experience and potential motion sickness, even with low server latency, as predictive streaming methods often fall short in precision, especially for non-linear camera movements.
Innovation Solution
The system modifies rendered content at a client device based on a current camera view by updating a parallax pixel map, which includes an image and a height map, to generate a current display frame, allowing quick response to user inputs with minimal latency while leveraging server rendering resources, thus providing an immersive experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If predictive streaming is used to reduce latency, then response time to user input is improved, but precision of camera movement prediction deteriorates, especially for non-linear movements
Solution Approach 1:
The server renders frames in advance based on predicted camera movements and transmits them before the actual camera movement occurs. The client device then adjusts these pre-rendered frames using parallax occlusion rendering to account for the actual camera position, combining preliminary action with real-time correction to achieve both low latency and high precision.
2Productivity
If frame rate and responsiveness requirements are increased for VR games, then user experience is improved, but graphics complexity and detail are reduced
Solution Approach 1:
The rendering process is segmented between server and client devices. The server performs computationally intensive tasks such as scene rendering and generates detailed graphics, while the client device performs lighter parallax occlusion rendering adjustments. This division allows high frame rates with maintained graphics quality by leveraging the server's rendering power.
3Manufacturing precision
If server rendering power is used for detailed graphics, then graphics quality is improved, but latency increases due to network transmission time
Solution Approach 1:
The server pre-renders detailed graphics frames in advance and transmits them to the client before the user's actual camera movement occurs. The client then quickly adjusts these pre-rendered frames using parallax occlusion rendering, eliminating the need for real-time server rendering and reducing latency while maintaining high graphics quality.
4Device complexity
If client device has limited rendering power, then device complexity is reduced, but rendering capability deteriorates
Solution Approach 1:
The client device receives pre-rendered frames from the server and creates a simplified representation called a parallax pixel map that stores only essential depth information. This copying approach allows the client to perform lightweight parallax occlusion adjustments without needing powerful rendering hardware, maintaining both low device complexity and adequate rendering capability.
Data Source
AI summary
A server provides image data including detailed geometry and shading information for one of more objects in a scene from a last known camera orientation and placement (a “first camera view”) and a height map indicating a distance from the first camera view to each pixel of the image. The image data and the height map are collectively referred to as a “parallax pixel map”. A client device receives the parallax pixel map from the server and updates the parallax pixel map based on a current camera orientation and placement (a “first camera view”). The client device projects the updated parallax pixel map onto the image of the scene based on the current camera view to generate a current display frame. The client device then provides the current display frame for display.