Foveated Rendering for VR Video Playback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
VR and AR game streaming experiences are hindered by the passive nature of viewers, who cannot control the viewpoint, leading to nausea and frustration due to the broadcaster's head and eye movements being tracked, rather than the viewer's.
Innovation Solution
A method and system for video recording and playback that employs gaze tracking and foveated rendering, allowing the viewer to freely look around a spherical image while maintaining high resolution only in the viewed area, with lower resolution in peripheral regions, and optionally extending this to a full sphere for immersive experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the recorded images track the broadcaster's head and eye movements, then the broadcaster's viewpoint is accurately captured, but the viewer experiences nausea and frustration due to inability to control their own viewpoint
Solution Approach 1:
The system dynamically switches between two operational modes: a first mode where the viewpoint tracks the broadcaster's head and eye movements for accurate representation, and a second mode where the viewpoint remains fixed or allows viewer control for comfort. This dynamic adaptability resolves the contradiction by allowing the system to optimize for either accuracy or ease of operation depending on the current viewing context.
2Manufacturing precision
If the entire spherical image is rendered at high resolution, then the visual quality is improved, but the computational overhead increases significantly
Solution Approach 1:
The system applies different rendering qualities to different regions of the spherical image based on the viewer's current viewpoint. The central region corresponding to the active viewpoint is rendered at high resolution, while peripheral regions are rendered at lower resolution. This local differentiation maintains visual quality where it matters most while significantly reducing the overall computational overhead.
Solution Approach 2:
Instead of rendering the entire spherical image at high resolution, the system performs partial rendering only for the portion of the sphere that corresponds to the viewer's current field of view. This partial action approach provides sufficient visual quality for the active viewing area while avoiding the excessive computational cost of rendering all regions at maximum resolution.
Data Source
AI summary
A method includes: rendering a field of view of a virtual environment for a first user of a head mounted display, rendering a view of the virtual environment outside the field of view of the first user at a lower resolution, outputting the rendered field of view for display to the first user, and recording the combined render as a video for subsequent viewing by a second user. A corresponding method includes requesting to download or stream the video from a remote source, receiving the download or stream of the video from the remote source, and outputting at least part of the stream or video for display to a second user wearing a head mounted display.