HMD Reprojection for Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Head-mounted displays (HMDs) experience a timing inconsistency between detecting the user's head attitude and displaying the corresponding image, leading to user discomfort due to delayed image updates and partial pixel display, which affects immersion in virtual reality experiences.
Innovation Solution
An information processing system that includes an HMD and an image generating method using reprojection techniques to align image display with the user's head attitude, reducing latency and ensuring simultaneous display of all pixels by predicting future attitudes and adjusting pixel positions accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the HMD updates the display screen in a linked manner with the user's head attitude, then the sense of immersion in the visual world is enhanced, but timing inconsistency occurs between the detected attitude and the displayed image
Solution Approach 1:
The system performs preliminary actions by detecting the head attitude in advance and pre-processing the image data before display. By preparing the transformed image data ahead of time based on predicted head movements, the system reduces the timing gap between attitude detection and image display, thereby maintaining immersion while minimizing latency.
Solution Approach 2:
The system dynamically adjusts the display updates based on the user's head movement characteristics. By continuously monitoring head attitude changes and adapting the image transformation parameters in real-time, the system optimizes the balance between maintaining immersion through frequent updates and reducing timing inconsistency by adjusting update frequency based on actual movement dynamics.
2Ease of manufacture
If the HMD displays pixels successively from the top scanning line down, then the display process is simplified, but the attitude of the HMD may change during display causing user discomfort
Solution Approach 1:
The system performs preliminary actions by predicting the head attitude at the time when each scanning line will be displayed and pre-transforming the image data accordingly. This allows the display process to remain simple and sequential while compensating for head movements that occur during the display process, thereby eliminating user discomfort without complicating the display mechanism.
Solution Approach 2:
The system applies different transformation parameters to different regions of the image corresponding to different scanning lines. By adjusting the image transformation locally for each line based on predicted head attitude at the time of display, the system maintains the simplicity of sequential display while compensating for head movements, thus preventing user discomfort.
3Measurement precision
If the image is generated based on the detected attitude of the HMD, then the image reflects the current head position, but latency occurs between detection and display
Solution Approach 1:
The system performs preliminary actions by detecting and predicting head attitude in advance and pre-processing the image transformation before actual display. By preparing the transformed image based on predicted head movements, the system reduces latency while maintaining image accuracy, as the transformation is already computed and ready for immediate display without waiting for the full detection-to-display cycle to complete.
Solution Approach 2:
The system dynamically adjusts the image generation process by continuously updating the transformation parameters based on real-time head attitude detection and prediction. This dynamic approach allows the system to maintain high image accuracy by always using the most current head position data while reducing latency through optimized processing that adapts to the actual timing and characteristics of head movements.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
An information processing apparatus 10 generates a reference image including information to be presented to a user in accordance with a first attitude of an HMD 100 identified at a first point in time. Based on the difference between the first attitude and a second attitude of the HMD 100 identified at a second point in time after the first point in time, the information processing apparatus 10 generates from the reference image a display image to be displayed on the HMD 100. Upon generation of the display image, the later the timing for displaying a specific image in the display image displayed on the HMD 100, the larger the amount of displacement is made by the information processing apparatus 10 between the position of the specific pixel in the display image and the position of a corresponding image in the reference image serving as the source from which to set the pixel value of the specific pixel.