3D Frame Extrapolation Using Application Motion Vectors

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional time warp solutions in artificial reality systems fail to account for translational movement and do not address animation stutters, relying on lower quality motion vectors estimated from two-dimensional images, which limits the ability to provide a high-quality, high-resolution experience on mobile head-mounted displays due to computing power constraints.

Innovation Solution

A novel frame extrapolation method that uses application-generated three-dimensional motion vectors and depth maps to calculate and reproject object positions, accounting for both rotational and translational movements, allowing for high-quality frame extrapolation and reprojection on mobile devices.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional time warp solutions are used to correct rotational movement, then the system can accommodate user viewpoint changes, but it fails to correct translational movement and animation stutters

Engineering Contradiction:

Improveviewpoint change accommodationVSAvoidmovement correction accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent extends traditional 2D image rotation to 3D spatial transformation by incorporating depth information. Motion vectors are applied in three-dimensional space rather than just on the image plane, enabling correction of both rotational and translational movements. The system transforms images through 3D warping based on depth maps and 3D motion vectors, resolving the limitation of 2D-only correction.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The system performs frame extrapolation by predicting future frame content based on current and past frames before actually rendering them. Motion vectors and depth information are used to pre-calculate transformations, allowing the system to prepare corrected frames in advance and reduce latency while maintaining accurate movement correction.

Inventive Principle:

Principle #10Preliminary action

2Productivity

If traditional frame extrapolation uses motion vectors estimated from two-dimensional images, then the system can generate future frames, but the motion vectors are lower quality and do not account for 3D movement

Engineering Contradiction:

Improveframe generation capabilityVSAvoidmotion vector accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent introduces depth maps as an intermediary element between 2D images and 3D motion vectors. Depth information serves as a mediator that enables the system to extract accurate 3D motion vectors from 2D image sequences. The depth map provides the spatial context needed to convert 2D pixel movements into accurate 3D motion vectors, significantly improving measurement precision.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system transitions from estimating 2D motion vectors to extracting 3D motion vectors by incorporating depth information. This dimensional extension allows the system to capture true three-dimensional movement patterns, improving the accuracy and quality of motion vectors used in frame extrapolation.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Manufacturing precision

If mobile HMD renders high resolution frames at high frame rate, then the user experience quality improves, but computing power requirements become excessive for mobile devices

Engineering Contradiction:

Improverendering qualityVSAvoidcomputing power requirement

Core Design Contradiction:

Manufacturing precisionVSPower

Solution Approach 1:

The system performs frame extrapolation and reprojection calculations in advance before the actual frame needs to be displayed. By pre-calculating motion vectors, depth transformations, and image warping operations, the system reduces the computational burden during real-time rendering while maintaining high frame rates and quality.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of rendering every frame at full resolution from scratch, the system uses frame extrapolation to generate future frames partially based on current and past frames. This partial rendering approach significantly reduces computational requirements while maintaining acceptable quality, allowing mobile devices to achieve high frame rates without excessive power consumption.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11783533B2Frame extrapolation with application generated motion vector and depth

Publication Date: 2023.10.10 META PLATFORMS TECHNOLOGIES LLC

AI summary

In one embodiment, a method includes receiving a rendered image, motion vector data, and a depth map corresponding to a current frame of a video stream generated by an application, calculating a current three-dimensional position corresponding to the current frame of an object presented in the rendered image using the depth map, calculating a past three-dimensional position of the object corresponding to a past frame using the motion vector data and the depth map, estimating a future three-dimensional position of the object corresponding to a future frame based on the past three-dimensional position and the current three-dimensional position of the object, and generating an extrapolated image corresponding to the future frame by reprojecting the object presented in the rendered image to a future viewpoint associated with the future frame using the future three-dimensional position of the object.