Frame Rendering Extrapolation for Head-Mounted Displays

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Conventional frame rendering technologies require high-end hardware to achieve high frame rates for fluent virtual reality experiences, leading to increased costs and inefficiencies.

Innovation Solution

A frame rendering apparatus and method for head-mounted devices that includes a storage module, motion sensor, and processing module to generate extrapolated frames by performing motion estimation and calibration on consecutive input frames, allowing for high frame-rate display without additional hardware costs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional frame rendering technology is used, then hardware cost is reduced, but frame rate is insufficient for fluent virtual reality experience

Engineering Contradiction:
Improveframe rateVSAvoidhardware cost
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The patent applies preliminary action by performing motion estimation and calibration on consecutive input frames before display. The system pre-processes frames to generate extrapolated frames that predict future frame positions, allowing the display to present high-frame-rate content without requiring high-end rendering hardware at the moment of display. This advance preparation resolves the contradiction by shifting computational load from real-time rendering to pre-processing.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent uses copying by generating extrapolated frames that are copies or predictions of future frame states based on motion estimation from previous frames. Instead of rendering each frame independently in real-time, the system creates predicted copies of what future frames will look like, enabling high frame rates through intelligent frame generation rather than brute-force rendering power.

Inventive Principle:
Principle #26Copying

2Reliability

If top-tier hardware setup is used, then fluent virtual reality experience is achieved, but hardware cost increases

Engineering Contradiction:
Improvefluent experienceVSAvoidhardware cost
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent applies parameter changes by transforming the frame rendering approach from direct real-time rendering to a parameter-based prediction system. The system changes the fundamental parameter from rendering every frame at full quality to using motion vectors and calibration data to generate intermediate frames. This parameter transformation maintains fluent experience while dramatically reducing hardware requirements.

Inventive Principle:
Principle #35Parameter changes

Solution Approach 2:

The patent substitutes the mechanical rendering system with a computational prediction system. Instead of relying on raw rendering power to generate each frame, the system uses motion estimation algorithms and calibration data to mathematically predict frame positions and generate extrapolated frames. This substitution replaces brute-force mechanical rendering with intelligent computational methods, reducing hardware cost while maintaining fluency.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

3Measurement precision

If motion estimation is performed on all pixel values, then frame extrapolation accuracy is improved, but processing time increases

Engineering Contradiction:
Improveframe extrapolation accuracyVSAvoidprocessing time
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The patent applies segmentation by dividing the image into different layers (first image layer and second image layer) and performing motion estimation separately on each layer. This segmentation allows the system to process only relevant pixel values for motion estimation while handling other layers differently, thereby improving extrapolation accuracy for motion-prone regions without unnecessarily processing all pixels, thus balancing accuracy with processing time efficiency.

Inventive Principle:
Principle #1Segmentation

Data Source

PatentEP3451127B1Frame rendering apparatus, method and non-transitory computer readable storage medium
Publication Date: 2020.10.21 HTC CORP
  • EP3451127B1 patent drawingFigure 1~2
  • EP3451127B1 patent drawingFigure 3
  • EP3451127B1 patent drawingFigure 4

AI summary

A frame rendering method used in a head-mounted device that includes the steps outlined below. Input frames are received. First and second orientation information corresponding to a first and a second input frames are retrieved from a motion sensor. Predicted orientation information of a predicted orientation corresponding to a target time spot is generated according to the first and the second orientation information. Orientation calibration is performed on the first and the second input frames according to the first and the second orientation information to respectively generate a first and a second calibrated frames corresponding to the predicted orientation. One of a plurality of extrapolated frames corresponding to the target time spot is generated according to the first calibrated frame and the second calibrated frame.