Frame Region Interpolation for HMD Latency Compensation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In head-mounted display systems for augmented and virtual reality applications, existing technologies face challenges in accurately displaying images due to latency in the rendering process, where the user's head orientation changes between the start of rendering and the actual display, leading to misalignment of images with the user's current view.

Innovation Solution

A method is introduced where the display processor transforms frames based on predicted view orientations by interpolating between initial and later predicted view orientation transformation matrices for different regions of a frame, allowing for accurate display alignment with the user's current head position during the scanning process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If frames are rendered based on head orientation data at the beginning of rendering, then the rendering process can be completed, but the displayed image becomes misaligned with the user's current head position due to latency

Engineering Contradiction:

Improveimage alignment accuracyVSAvoidrendering latency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The transformation engine performs preliminary transformation of frame regions using predicted view orientations before the complete frame is ready for display. By transforming regions based on predicted head positions rather than waiting for complete rendering, the system compensates for latency and maintains accurate image alignment with the user's current head position.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts transformation matrices for different frame regions based on predicted view orientations. Instead of using a single static transformation for the entire frame, the transformation engine applies region-specific transformations that adapt to the predicted head position, allowing the display to remain accurate despite rendering delays.

Inventive Principle:

Principle #15Dynamics

2Device complexity

If the entire frame is transformed using a single predicted view orientation, then the processing is simple, but the transformation accuracy decreases for regions displayed at different times

Engineering Contradiction:

Improveprocessing complexityVSAvoidtransformation accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The transformation engine divides the frame into multiple regions and applies separate transformation matrices to each region based on its specific display timing. This segmentation allows each region to be transformed with the appropriate predicted view orientation for its display time, improving overall transformation accuracy while maintaining manageable processing complexity through systematic region handling.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different transformation qualities are applied to different regions of the frame based on their display characteristics. Regions are transformed using locally appropriate predicted view orientations that match their specific display timing, rather than applying a uniform transformation across the entire frame. This local quality approach optimizes accuracy for each region's specific display context.

Inventive Principle:

Principle #3Local quality

3Measurement precision

If region-specific predicted view orientation transformations are applied to all regions, then the image alignment accuracy improves, but the computational load increases

Engineering Contradiction:

Improvedisplay alignment accuracyVSAvoidcomputational load

Core Design Contradiction:

Measurement precisionVSPower

Solution Approach 1:

The transformation engine applies full region-specific transformations only to regions that require high precision alignment, while using simplified or interpolated transformations for other regions. This partial action approach maintains necessary accuracy for critical display regions while reducing overall computational load by avoiding excessive processing in regions where full precision is not required.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system uses interpolated predicted view orientations as intermediary values between fully computed region-specific transformations. By interpolating transformation parameters for regions between fully processed anchor regions, the system achieves acceptable accuracy for intermediate regions with significantly reduced computational effort, balancing precision and processing load.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS10692420B2Data processing systems

Publication Date: 2020.06.23 ARM LTD

AI summary

When displaying a frame that is to be transformed based on a predicted view orientation, the frame is transformed based on a predicted view orientation by, for each of plural different regions of the frame, determining a predicted view orientation transformation matrix to use to transform the region of the frame by interpolating between a first predicted view orientation matrix that corresponds to a transformation for a predicted view orientation for a first time during the display of the frame and a second predicted view orientation matrix that corresponds to a predicted view orientation for a second, later time during the display of the frame; and then using the interpolated predicted view orientation transformation matrix determined for the region of the frame to transform the region of the frame. In this way each different region of the frame is subjected to a different predicted view orientation transformation.