Client Video Slice Rendering with Overlapping Decode and Display
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud gaming systems experience significant latency due to differences in clock frequencies and phase alignment between cloud gaming servers and clients, leading to instability and increased latency in video frame transmission and display.
Innovation Solution
Synchronize and offset VSYNC signals between the cloud gaming server and client to align their frequencies and adjust timing offsets, allowing for reduced latency by overlapping decode and display operations, and dynamically adjusting client buffering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If video frames are transmitted and displayed in sequential order waiting for complete reception and decoding, then display stability is maintained, but one-way latency increases
Solution Approach 1:
The system performs preliminary actions by receiving and decoding video frames in advance before they are needed for display. The client receives encoded video frames from the server, decodes them into decoded slices, and prepares them for display before the actual display time arrives. This advance preparation reduces the latency experienced during actual display operations.
Solution Approach 2:
The system maintains continuous useful action by overlapping the reception, decoding, and display operations. Instead of waiting for one operation to complete before starting the next, the system continuously receives encoded frames, decodes them as they arrive, and displays decoded slices as soon as they are ready, maintaining a continuous pipeline of operations that minimizes idle time and reduces overall latency.
2Manufacturing precision
If the client waits for complete reception of encoded video frames before decoding and display, then decoding accuracy is ensured, but processing time increases
Solution Approach 1:
The video frame is divided into multiple encoded slices that can be received and decoded independently. Instead of waiting for the entire frame to be received before decoding, the system decodes individual slices as they are received from the network. This segmentation allows parallel processing of different parts of the frame, reducing overall processing time while maintaining decoding accuracy for each slice.
Solution Approach 2:
The system performs partial decoding actions by decoding only the portions of video frames that have been received so far, rather than waiting for complete reception. The client displays decoded slices as soon as they are decoded, even if the entire frame is not yet received. This partial action approach reduces processing time by avoiding idle waiting periods while maintaining sufficient quality for display.
3Adaptability or versatility
If VSYNC signals from server and client are not synchronized, then operational independence is maintained, but timing stability deteriorates
Solution Approach 1:
The system uses feedback mechanisms to synchronize VSYNC signals between server and client. The client monitors the timing of received video frames and adjusts its VSYNC signal accordingly to match the server's timing. This feedback loop ensures that both server and client operate in synchronization, maintaining timing stability while allowing each to maintain operational independence in their respective processing pipelines.
Solution Approach 2:
The system adjusts timing parameters of VSYNC signals to achieve synchronization between server and client. By dynamically changing the timing parameters of VSYNC signals based on measured latency and timing offsets, the system maintains synchronization while allowing flexibility in operational timing. This parameter adjustment enables the system to adapt to varying network conditions while maintaining stable timing relationships.
Data Source
AI summary
A method of cloud gaming is disclosed. The method including receiving an encoded video frame at a client, wherein a server executes an application to generate a rendered video frame which is then encoded at an encoder at the server as the encoded video frame, wherein the encoded video frame includes one or more encoded slices that are compressed. The method including decoding the one or more encoded slices at a decoder of the client to generate one or more decoded slices. The method including rendering the one or more decoded slices for display at the client. The method including begin displaying the one or more decoded slices that are rendered before fully receiving the one or more encoded slices at the client.