AR Frame Projection Using Motion Vector Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In augmented reality environments, there is a challenge in maintaining the alignment and stability of virtual objects with the real-world environment, leading to perceived drift due to latency in displaying images of world-locked virtual objects, which affects the user experience and requires high rendering rates to reduce power consumption.
Innovation Solution
The use of motion vector estimation to generate and display images at a frame rate greater than the rendering frame rate, incorporating late stage graphical adjustments and homographic transformations to simulate predicted poses and remove camera movement, allowing for accurate and efficient projection of virtual objects within the AR environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high rendering rates are used to reduce latency and improve virtual object stability, then user experience is improved, but power consumption increases
Solution Approach 1:
The system performs preliminary actions by predicting future device poses and pre-rendering frames at lower rates before they are actually needed for display. This allows the display to show pre-rendered content with minimal latency without requiring continuous high-rate rendering, thus reducing power consumption while maintaining virtual object stability.
Solution Approach 2:
The system creates copies of rendered frames and applies transformations to these copies rather than re-rendering the entire scene at high rates. By copying and transforming previously rendered frames with adjusted pose information, the system maintains visual accuracy and stability while avoiding the high computational cost and power consumption of full re-rendering.
2Loss of time
If frame re-rendering is performed at high rates to reduce latency, then virtual object alignment is improved, but processing time increases
Solution Approach 1:
The system performs preliminary frame rendering and transformation operations in advance, preparing display content before it is needed. By predicting future poses and pre-processing frames ahead of time, the system reduces actual display latency without requiring continuous high-rate processing, thus improving processing efficiency.
Solution Approach 2:
Instead of re-rendering frames at high rates, the system copies previously rendered frames and applies pose transformations to the copies. This approach maintains the appearance of high-rate updating for low latency while avoiding the computationally expensive re-rendering process, thereby improving overall processing efficiency.
3Reliability
If motion vector estimation and homographic transformations are applied, then virtual object stability is improved, but device complexity increases
Solution Approach 1:
The system applies motion vector estimation and homographic transformations to copied frames rather than re-rendered frames. This approach maintains virtual object stability through accurate pose compensation while avoiding the need for complex re-rendering pipelines, thus reducing overall processing complexity.
Solution Approach 2:
The system replaces complex mechanical re-rendering operations with mathematical transformations (homographic transformations and motion vector estimations) applied to existing frames. This substitution of mathematical operations for computational rendering significantly reduces processing complexity while maintaining virtual object stability.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Motion vector estimation is provided for generating and displaying images at a frame rate that is greater than a rendering frame rate. The displayed images may include late stage graphical adjustments of pre-rendered scenes that incorporate motion vector estimations. A head-mounted display (HMD) device may determine a predicted pose associated with a future position and orientation of the HMD, render a current frame based on the predicted pose, determine a set of motion vectors based on the current frame and a previous frame, generate an updated image based on the set of motion vectors and the current frame, and display the updated image on the HMD. In one embodiment, the HMD may determine an updated pose associated with the HMD subsequent to or concurrent with generating the current frame, and generate the updated image based on the updated pose and the set of motion vectors.