Frame Interpolation Using Motion Vector Projection

Solution Overview

Problem

Existing frame-interpolation methods in imaging technologies, such as those used in cameras, gaming, VR, and AI, require multiple network inferences for each time step, leading to high power consumption and latency, which is inefficient for real-time applications.

Innovation Solution

A method that uses a single network inference to estimate motion vectors and occlusion masks, followed by a low-complexity projection algorithm to generate interpolated frames at arbitrary time steps, reducing power and latency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If multiple network inferences are performed for each time step in frame interpolation, then motion estimation accuracy is improved, but power consumption increases

Engineering Contradiction:

Improvemotion estimation accuracyVSAvoidpower consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent segments the frame interpolation process into two distinct stages: (1) a machine learning-based motion estimator that performs accurate motion vector estimation, and (2) a projection algorithm that generates intermediate frames using the estimated motion vectors. This segmentation allows the computationally intensive ML model to be executed only once per frame pair rather than repeatedly for each intermediate frame, thereby maintaining motion estimation accuracy while significantly reducing power consumption.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies preliminary action by performing motion vector estimation before generating intermediate frames. The motion estimator first computes motion vectors between input frames, and these pre-computed vectors are then reused by the projection algorithm to generate multiple intermediate frames at different time steps. This preliminary computation avoids redundant ML inferences and reduces overall power consumption.

Inventive Principle:

Principle #10Preliminary action

2Manufacturing precision

If multiple network inferences are performed for each time step in frame interpolation, then interpolation quality is improved, but latency increases

Engineering Contradiction:

Improveinterpolation qualityVSAvoidlatency

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent segments the frame interpolation process into two distinct stages: (1) a machine learning-based motion estimator that performs accurate motion vector estimation, and (2) a projection algorithm that generates intermediate frames using the estimated motion vectors. This segmentation allows the computationally intensive ML model to be executed only once per frame pair rather than repeatedly for each intermediate frame, thereby maintaining motion estimation accuracy while significantly reducing power consumption.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies preliminary action by performing motion vector estimation before generating intermediate frames. The motion estimator first computes motion vectors between input frames, and these pre-computed vectors are then reused by the projection algorithm to generate multiple intermediate frames at different time steps. This preliminary computation avoids redundant ML inferences and reduces overall power consumption.

Inventive Principle:

Principle #10Preliminary action

3Use of energy by moving object

If a single network inference is used for motion estimation, then power consumption is reduced, but motion vector accuracy may deteriorate

Engineering Contradiction:

Improvepower consumptionVSAvoidmotion vector accuracy

Core Design Contradiction:

Use of energy by moving objectVSMeasurement precision

Solution Approach 1:

The patent introduces a projection algorithm as an intermediary between the motion estimator and frame generation. The ML-based motion estimator provides initial motion vector estimates, and the projection algorithm refines these vectors by projecting them to intermediate time steps and generating corresponding intermediate frames. This intermediary step ensures that motion vector accuracy is maintained while avoiding redundant ML inferences, thus reducing power consumption.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces repeated mechanical ML inferences with a mathematical projection algorithm for generating intermediate frames. Instead of running the ML model multiple times for each intermediate frame, the system uses projection-based interpolation that leverages the initial motion estimates. This substitution maintains accuracy while dramatically reducing computational power requirements.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20260075160A1Generating interpolated image data

Publication Date: 2026.03.12 QUALCOMM INC

AI summary

Systems and techniques are described herein for interpolating image data. For instance, a method for interpolating image data is provided. The method may include processing a first image frame and a second image frame using a motion estimator to generate first motion vectors, wherein the motion estimator comprises a machine-learning model trained to generate motion vectors based on image frames; projecting the first motion vectors to generate second motion vectors; and generating a third image frame based on the first image frame, the second image frame, and the second motion vectors.