Virtual Reference Frame Encoding for Low-Latency Device-Cloud Rendering

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing rendering technologies face high interaction delays and inefficient bit rate overheads due to the reliance on cloud-side servers for image/video processing, which is exacerbated by increasing image quality demands and device computational limitations.

Innovation Solution

A device-cloud collaboration system where the terminal device performs rendering and encoding, using a virtual reference frame to predict and encode residual images, reducing the need for intermediate results from the server, thereby minimizing interaction delay and bit rate overheads.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If cloud-side server performs rendering and transmits intermediate rendering results, then rendering quality is improved, but interaction delay increases and bit rate overhead increases

Engineering Contradiction:

Improverendering qualityVSAvoidinteraction delay

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent extracts the intermediate rendering result transmission step from the conventional cloud rendering pipeline. By having the terminal device generate and use intermediate rendering results locally without requiring the cloud server to transmit them, the system eliminates the transmission delay associated with sending these intermediate results while maintaining the quality benefits of multi-pass rendering.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The terminal device performs self-service by generating its own intermediate rendering results locally during the rendering process. This eliminates the need for the terminal to wait for and receive intermediate results from the cloud server, thereby reducing interaction delay while still achieving high rendering quality through local computational capabilities.

Inventive Principle:

Principle #25Self-service

2Manufacturing precision

If cloud-side server transmits intermediate rendering results, then rendering quality is improved, but bit rate overhead increases

Engineering Contradiction:

Improverendering qualityVSAvoidbit rate overhead

Core Design Contradiction:

Manufacturing precisionVSQuantity of substance

Solution Approach 1:

The patent removes the transmission of intermediate rendering results from the cloud server to terminal device pipeline. By extracting this transmission step entirely, the system eliminates the bit rate overhead associated with sending these intermediate results, while the terminal device still achieves high rendering quality through local intermediate result generation.

Inventive Principle:

Principle #2Taking out (Extraction)

3Loss of time

If terminal device performs rendering locally, then interaction delay is reduced, but computational power requirements increase

Engineering Contradiction:

Improveinteraction delayVSAvoidcomputational power

Core Design Contradiction:

Loss of timeVSPower

Solution Approach 1:

The patent applies partial action by having the terminal device perform only the intermediate rendering steps locally rather than the complete rendering pipeline. The cloud server still performs the final rendering and compression, so the terminal device's computational burden is reduced to manageable levels while still achieving the delay reduction benefits of local processing.

Inventive Principle:

Principle #16Partial or excessive action

4Productivity

If conventional encoding is used without virtual reference frames, then encoding simplicity is maintained, but encoding efficiency decreases

Engineering Contradiction:

Improveencoding efficiencyVSAvoidencoding complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent generates virtual reference frames in advance during the rendering process, before the actual encoding step. These pre-generated virtual reference frames serve as efficient prediction references during encoding, significantly improving encoding efficiency by reducing the complexity of the encoding process itself, while the overall system complexity remains manageable due to the automated generation of these reference frames.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20250324091A1Device-cloud collaboration system, encoding and decoding method, and electronic device

Publication Date: 2025.10.16 HUAWEI TECH CO LTD

AI summary

Embodiments of this application provide a device-cloud collaboration system, an encoding and decoding method, and an electronic device. The encoding method includes: performing rendering processing on a three-dimensional scene based on a rendering parameter, to obtain a rendered image, where the rendering parameter includes a first rendering parameter obtained from a terminal device; selecting a first intermediate rendering result based on an intermediate rendering result generated in a rendering processing process; generating a virtual reference frame based on the first intermediate rendering result; predicting the rendered image based on the virtual reference frame, to obtain a predicted image; and encoding a residual image between the predicted image and the rendered image, and encoding encoded data of the residual image into a bitstream. The bitstream does not include encoded data of the first intermediate rendering result.