Temporal Frame Transformation for Low Latency Application Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Client devices, particularly mobile devices, face challenges in providing low latency for interactive applications due to the limitations of their hardware and the high cost and limited availability of datacenter spaces in metropolitan areas, leading to undesirable latency in remote rendering solutions.
Innovation Solution
A system that offloads high-performance graphics rendering to a remote server and an edge server, where the remote server generates a resolution delta frame indicating differences between high and low resolution frames, allowing the edge server to transform low resolution frames into high resolution frames, reducing bandwidth requirements and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If rendering is offloaded to a remote server, then graphical detail and processing capability are improved, but latency increases due to geographical distance
Solution Approach 1:
The rendering process is segmented into two parts: low-resolution frame generation (performed locally at edge server to maintain low latency) and high-resolution detail enhancement (performed at remote server to improve graphical detail). This segmentation allows each component to operate optimally without being constrained by the other's limitations.
Solution Approach 2:
A resolution delta frame is introduced as an intermediary data structure that captures only the differences between low and high resolution frames. This intermediary representation significantly reduces bandwidth requirements and transmission time, thereby reducing latency while still enabling high-resolution rendering enhancement.
2Manufacturing precision
If high-resolution rendering is performed at remote server, then visual quality is improved, but bandwidth requirements increase
Solution Approach 1:
Instead of transmitting complete high-resolution frames, the system extracts only the essential difference information (resolution delta frame) that represents the visual improvements. This extracted differential data is much smaller in size, significantly reducing bandwidth requirements while preserving visual quality enhancement.
Solution Approach 2:
The system changes the parameter representation from full high-resolution frame data to differential delta frame data. This parameter transformation reduces the data volume by focusing only on the changes needed to achieve high resolution, thereby reducing bandwidth consumption.
3Productivity
If multiple servers are used for collaborative rendering, then rendering performance is improved, but system complexity increases
Solution Approach 1:
The system merges the rendering capabilities of remote and edge servers into a unified collaborative process. The edge server handles time-critical low-resolution rendering while the remote server handles quality-critical high-resolution enhancement, combining their strengths to achieve high rendering performance without requiring each server to independently perform all rendering tasks.
Solution Approach 2:
The resolution delta frame mechanism serves multiple functions: it acts as a communication protocol between servers, a data compression format, and a quality enhancement method. This multi-functional approach reduces system complexity by using a single versatile mechanism rather than multiple specialized components.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
The present describes low latency streaming using temporal frame transformation. An execution component in an edge server executes a first instance of an application. A server interface component receives, from a remote server, a resolution delta frame indicating differences between a high resolution first frame and a low resolution first frame of a second instance of the application or, alternatively, receives the high resolution first frame. A video manipulation component generates a motion delta frame by identifying differences between a low resolution first frame and a low resolution second frame of the first instance of the application. The video manipulation component generates a high resolution transformed frame by applying the resolution delta frame and the motion delta frame to the low resolution second frame.