Neural Network Temporal Upsampling via Depth-Aware Warping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current techniques for temporal upsampling of image frames in graphics and video applications often result in synthesized frames with limited quantization and accuracy, failing to effectively smooth motion and increase frame rate without significant computational overhead.

Innovation Solution

The implementation of a neural network-based system that uses motion vector interpolation, warping, and depth-aware warping to generate temporally upscaled image frames by blending and combining warped frames with residual values, employing a U-Net architecture for improved accuracy and efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If traditional temporal upsampling techniques are used, then frame rate can be increased, but quantization accuracy and motion smoothness deteriorate

Engineering Contradiction:

Improveframe rateVSAvoidquantization accuracy

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The patent replaces traditional mechanical interpolation methods with a neural network-based system that uses U-Net architecture, motion vector interpolation, and depth-aware warping to generate synthesized frames with high quantization accuracy while maintaining increased frame rate

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system changes the approach from simple frame interpolation to a multi-parameter process involving motion vectors, depth information, and neural network predictions, enabling both high frame rate and high accuracy simultaneously

Inventive Principle:

Principle #35Parameter changes

2Productivity

If traditional temporal upsampling techniques are used, then frame rate can be increased, but motion smoothness deteriorates

Engineering Contradiction:

Improveframe rateVSAvoidmotion smoothness

Core Design Contradiction:

ProductivityVSStability of the object's composition

Solution Approach 1:

The patent replaces traditional mechanical interpolation with neural network-based synthesis using U-Net architecture and depth-aware warping, producing motion smoothness that traditional methods cannot achieve at high frame rates

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system introduces motion vectors and depth information as intermediary elements between source frames, using these mediators to guide the neural network in generating smooth transitions that maintain motion consistency

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If traditional temporal upsampling techniques are used, then frame rate can be increased, but computational resources required increase significantly

Engineering Contradiction:

Improveframe rateVSAvoidcomputational resources

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the upsampling process into distinct components: motion vector interpolation, depth-aware warping, and neural network prediction, allowing each component to be optimized independently and reducing overall computational burden

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system substitutes computationally expensive traditional interpolation with a neural network model that, once trained, can generate frames efficiently, reducing real-time computational requirements while maintaining high frame rate and quality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20240029196A1System, devices and/or processes for temporal upsampling image frames

Publication Date: 2024.01.25 ARM LTD

AI summary

Example methods, apparatuses, and/or articles of manufacture are disclosed that may be implemented, in whole or in part, techniques to process image signal values sampled from a multi color channel imaging device. In particular, methods and/or techniques disclosed herein are directed to synthesizing a temporally upsampled image frame to be in a temporal sequence of images frames.