Cloud Gaming Livestream Sync Using Shared Audio-Video Timestamps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In traditional game livestreaming, there is a challenge in synchronizing audio-video streams between different terminals, particularly in cloud gaming scenarios where the performance requirements differ from traditional 2D game livestreaming, making direct livestreaming impossible.
Innovation Solution
A cloud gaming system where game rendering occurs on a cloud server, utilizing in-container rendering and media coding to synchronize audio-video streams, ensuring synchronization through shared timestamps for audio and video frames, and delivering synchronized streams to viewer terminals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If cloud gaming rendering is performed on cloud server, then performance consumption on client side is reduced, but audio-video synchronization complexity increases
Solution Approach 1:
The patent introduces a timestamp as an intermediary mechanism to synchronize audio and video streams. The cloud server generates consistent timestamps for both audio and video frames during rendering, and these timestamps are used by the client to synchronize playback without requiring complex synchronization logic on the client side. This mediator (timestamp) simplifies the synchronization process while enabling cloud-based rendering.
Solution Approach 2:
The cloud server performs preliminary synchronization actions by generating consistent timestamps for audio and video frames during the rendering process itself. This preliminary timestamp generation ensures that when the streams are transmitted to the client, they are already synchronized, reducing the need for complex client-side synchronization processing.
2Reliability
If real-time audio/video streaming is performed on mobile terminal, then audio-video synchronization is maintained, but device performance requirements increase
Solution Approach 1:
The patent extracts the heavy rendering and synchronization processing from the mobile terminal and relocates it to the cloud server. The mobile terminal only needs to receive and play back the pre-synchronized audio and video streams using the timestamps provided by the cloud server, significantly reducing device performance requirements while maintaining synchronization reliability.
3Adaptability or versatility
If cloud gaming livestreaming is implemented, then accessibility to high-performance games is improved, but audio-video synchronization reliability deteriorates
Solution Approach 1:
The cloud server performs preliminary synchronization by generating consistent timestamps for audio and video frames during the rendering process. This preliminary action ensures that synchronization is established before the streams are transmitted to the client, maintaining reliability even in cloud gaming livestreaming scenarios where network conditions may vary.
Solution Approach 2:
The patent implements a feedback mechanism where the client sends playback status information back to the cloud server, and the server adjusts the timestamp generation and stream transmission accordingly. This feedback loop ensures that audio-video synchronization is maintained reliably even when network conditions or client performance varies, enabling accessible cloud gaming livestreaming.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of this application disclose a game data processing method and apparatus, and a storage medium. The method includes: receiving a cloud gaming livestreaming request for a target cloud gaming transmitted by a cloud gaming client, and establishing a livestreaming service connection between a cloud server and a livestreaming server associated with the cloud gaming client based on the cloud gaming livestreaming request (S101); acquiring game image data and game audio data of the target cloud gaming via the livestreaming service connection, and recording an image collection timestamp of the game image data and an audio collection timestamp of the game audio data (S102); mixing, in response to acquiring to-be-processed audio data of a livestreamer object transmitted by the cloud gaming client, the game audio data and the to-be-processed audio data to obtain mixed audio data (S103); and obtaining a livestreaming data stream of the target cloud gaming based on the mixed audio data, the audio collection timestamp, the game image data and the image collection timestamp, and delivering the livestreaming data stream to a livestreaming client associated with the cloud gaming client through the livestreaming server (S104).