Separate Audio Video Streaming for Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Resource-constrained client devices struggle to render graphical user interfaces (GUIs) due to insufficient computing resources, and sending audio components with video frames can lead to delayed audio delivery, affecting user experience, as delays in audio delivery are more noticeable than video delays.
Innovation Solution
The method involves determining the playback position within a video content item, obtaining and transmitting audio data separately from video frames to the client device, ensuring timely audio delivery even if video frames are delayed, by using an edge device or cloud computing platform to provide audio data prior to video frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If audio components are sent together with video frames to resource-constrained client devices, then the system can maintain synchronized audio-video delivery, but audio delivery delays occur and are more noticeable to users
Solution Approach 1:
The patent segments the audio and video data streams by transmitting them through separate channels. Audio data is extracted and sent independently from video frames, allowing audio to be delivered and processed separately. This segmentation enables audio to arrive at the client device without being bottlenecked by video processing constraints, reducing audio delivery delays while maintaining synchronization through separate timing mechanisms.
Solution Approach 2:
The patent implements preliminary action by pre-processing and transmitting audio data before video frames. The system extracts audio components in advance, encodes them separately, and transmits them ahead of the corresponding video frames. This allows audio to be buffered and ready for playback before the video rendering process completes, ensuring timely audio delivery even on resource-constrained devices.
2Ease of operation
If audio decoding and re-encoding are performed on client devices, then audio can be processed locally, but additional computational resources are required
Solution Approach 1:
The patent applies the extraction principle by removing the audio decoding and re-encoding operations from the client device. Instead, audio data is extracted at the server side, transmitted in a processed format, and only final playback decoding is performed at the client. This extraction of computationally intensive operations significantly reduces the energy and computational resource requirements at the client device while still enabling local audio output.
Solution Approach 2:
The patent introduces an intermediary processing stage at the server side that handles audio decoding and re-encoding before transmission. This intermediary performs the heavy computational work centrally, then transmits the processed audio data to the client. The client device acts as a simple playback endpoint rather than a full processing node, reducing its computational burden while maintaining audio quality through the intermediary's processing.
Data Source
AI summary
In some implementations, a device includes one or more processors and a non-transitory memory. In some implementations, a method includes determining that a video frame to be presented at a client device includes a portion that corresponds to a particular playback position within a video content item. In some implementations, the method includes obtaining audio data that corresponds to the particular playback position within the video content item. In some implementations, the method includes transmitting the audio data to the client device separate from the video frame.