Real-time Video Coding via 3D Image Warping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Cloud gaming services face high bandwidth requirements due to inefficient real-time video encoding, limiting accessibility for users with lower bandwidth connections, as conventional H.264/AVC encoders lack optimizations that require momentary look-ahead buffers or multiple encoding passes.

Innovation Solution

A method that utilizes graphics rendering contexts, including pixel depth maps and camera motion, to select key frames and interpolate non-key frames through 3D image warping, encoding key frames and warping residues with H.264/AVC, dynamically allocating bit rates to improve coding efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If real-time H.264/AVC encoding is used in cloud gaming, then video streaming is enabled, but bandwidth requirements become excessively high

Engineering Contradiction:

Improvevideo encoding speedVSAvoidbandwidth consumption

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent segments video frames into key frames and non-key frames, applying different encoding strategies to each type. Key frames are encoded with higher quality and more bits, while non-key frames use warping residues with lower quality, reducing overall bandwidth consumption while maintaining real-time encoding capability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by selecting key frames and calculating warping residues before actual encoding. By pre-determining which frames need full encoding and which can be synthesized through warping, the system optimizes bandwidth allocation and reduces overall bandwidth requirements

Inventive Principle:

Principle #10Preliminary action

2Productivity

If conventional H.264/AVC encoder is used, then real-time encoding is achieved, but encoding efficiency is low due to lack of look-ahead buffer and multiple passes

Engineering Contradiction:

Improveencoding speedVSAvoidencoding efficiency

Core Design Contradiction:

ProductivityVSEase of manufacture

Solution Approach 1:

The patent introduces dynamic frame classification and selective encoding strategies. The encoder dynamically determines which frames should be encoded as key frames and which should use warping residues, adapting the encoding process to the specific characteristics of each frame rather than using a fixed conventional approach

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent introduces 3D image warping as an intermediary technique between the rendered frames and the final video encoding. This warping step acts as a mediator that enables efficient compression by creating synthesized frames that reduce the amount of data needing to be encoded, thereby improving encoding efficiency while maintaining real-time performance

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS8872895B2Real-time video coding using graphics rendering contexts

Publication Date: 2014.10.28 DEUTSCHE TELEKOM AG

AI summary

A method for encoding video for streaming includes receiving a plurality of sequential image frames generated by a 3D graphics rendering engine. Graphics rendering contexts are obtained, including pixel depth map, rendering camera parameters, and camera motion from the 3D rendering engine. The method next entails selecting key frames among the plurality of sequential image frames, interpolating non-key frames via 3D image warping, and encoding all key frames and warping residues of non-key frames. The system is implementable on a server linked to a mobile user device for receiving the encoded frame data. The mobile user device is configured to decode the encoded frame data and display a corresponding image to a user of the mobile user device.