Head Mounted Display Image Reprojection for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Head-mountable displays (HMDs) face challenges in providing an immersive experience by accurately rendering images in real-time, as they often suffer from latency issues due to the delay between detecting head movement and rendering the correct image, leading to potential user disorientation and nausea.
Innovation Solution
The technique involves capturing or rendering an image based on the initial view matrix and associating metadata with the image, which is then reprojected at display time to account for any differences between the initial and current head position, reducing latency by allowing the rendering operation to start earlier and adjusting the image in real-time to match the viewer's head movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the rendering operation waits to detect current head position before generating the image, then the image accuracy matches the viewer's head position, but the latency increases causing user disorientation and nausea
Solution Approach 1:
The system performs the rendering operation in advance using the initial view matrix before the head position is fully detected. The image is generated preliminarily and then reprojected later to account for head movement, allowing the rendering to start earlier and reduce latency while still achieving accurate final positioning through the reprojection step
2Loss of time
If the rendering operation starts earlier to reduce latency, then the latency is reduced improving user comfort, but the image may not match the current head position causing disorientation
Solution Approach 1:
The image processing is divided into two separate operations: (1) a rendering operation that generates the base image using the initial view matrix, and (2) a reprojection operation that adjusts the rendered image to account for head movement. This segmentation allows the rendering to occur earlier with reduced latency while the reprojection step ensures final image accuracy matches the current head position
Data Source
AI summary
An image processing method for a head mounted display device is provided, operable in respect of an image generated for display by the head mountable display device according to at least one of an initial position or orientation of a viewpoint. The method includes detecting one or both of a current position or orientation of the head mountable display device depending upon a display time at which the image is to be displayed. The method further includes processing the image using pixel position mappings in which at least a subset of pixels of the image are displaced by respective pixel displacements dependent upon a difference between at least one of the initial position and the detected current position, or the initial orientation and the detected current orientation at the time at which the image is to be displayed.