Audio Video Stream Synchronization via Timestamp Manipulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multimedia relay conferencing systems, synchronizing audio and video streams across multiple endpoints is complex, leading to challenges in maintaining lip-sync and requiring significant computing resources and bandwidth, especially when endpoints have different clock domains and lack support for audio relay codecs.
Innovation Solution
The technique involves manipulating timestamps of audio and video packets by calculating and applying a delta value to align them with the intermediate node's timing domain, allowing receiving endpoints to synchronize streams without relying on sender reports from each transmitting endpoint, thus reducing latency and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional audio relay conferencing is used to support multiple endpoints with different clock domains, then audio relay codec support is required at each endpoint, but this increases device complexity and computing resources needed
Solution Approach 1:
The patent introduces an intermediary entity (the audio relay server or gateway) that performs clock domain translation and audio relay functions. Instead of requiring each endpoint to support audio relay codecs directly, the intermediary handles the complex synchronization and codec conversion, allowing simple endpoints to participate in audio relay conferences without increasing their device complexity.
2Reliability
If sender reports from each transmitting endpoint are used for synchronization, then accurate lip-sync can be maintained, but this increases bandwidth consumption and processing overhead
Solution Approach 1:
The patent merges the synchronization function into a centralized process where a single sender report from the intermediary entity is used to synchronize all relayed audio streams. Instead of each transmitting endpoint sending individual sender reports, the intermediary consolidates the synchronization information into one report, reducing bandwidth consumption and processing overhead while maintaining synchronization accuracy across all endpoints.
3Productivity
If multiple audio streams are relayed with individual sender reports, then each stream can be synchronized independently, but this increases the number of synchronization packets and processing requirements at the intermediate node
Solution Approach 1:
The patent makes the single sender report from the intermediary entity universal for synchronizing multiple audio streams. The sender report contains synchronization information that applies to all relayed audio streams simultaneously, allowing the intermediate node to synchronize multiple streams without processing individual sender reports for each stream. This reduces the number of synchronization packets and processing requirements while maintaining the ability to synchronize independent streams.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A novel technique allows synchronizing a plurality of audio and video streams received at a receiving media relay endpoint (130) via an intermediate node (120), such as a media relay multipoint control unit. The received streams were generated by a plurality of transmitting media relay endpoints (130) and been relayed to the plurality of receiving media relay endpoints (130) via the intermediate node (120), but are seen as being using a single time domain, in terms of wall clock and manipulated timestamps, while preserving the correct capture time.