HLS Live Stream Synchronization via Encoding Timestamp Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Synchronizing playback of multiple HLS live media streams captured using separate cameras with different capture clocks is challenging due to the lack of direct correspondence between their presentation time stamps (PTSes), which are generated independently.
Innovation Solution
The technique involves receiving media streams with presentation time stamps (PTSes) and encoding time stamps (ETSes), determining correspondence between ETSes and PTSes for each stream, and using this correspondence to synchronize playback clocks across players, with the option to estimate and correct ETSes for segments without explicit timestamps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate HLS media encoders generate PTSes independently using separate capture clocks, then each stream can be encoded and streamed autonomously, but the PTSes in different streams do not correlate and synchronization becomes problematic
Solution Approach 1:
The patent introduces encoding timestamps (ETs) as an intermediary that both encoders can generate using a common reference (wall clock time). These ETs serve as a mediator between the independently running capture clocks, allowing the system to correlate timestamps across streams without requiring the capture clocks themselves to be synchronized.
Solution Approach 2:
The patent replaces reliance on mechanical capture clock synchronization with a software-based timestamp correlation system. Instead of requiring hardware clocks to be synchronized, the system uses ETs embedded in manifest files to establish temporal relationships between segments from different streams through computational processing.
2Ease of operation
If a media player uses a local player clock for playback timing, then playback can be controlled locally, but in live streams where reception rate cannot be controlled, playback rate must be adjusted based on buffer fullness measurements
Solution Approach 1:
The patent implements feedback mechanisms where the media player continuously monitors buffer fullness and adjusts playback rate accordingly. The system also uses timestamp information from received segments to provide feedback on expected playback timing, allowing the player to synchronize its clock with the live stream timeline while maintaining local control.
Solution Approach 2:
The patent makes the playback system dynamic by allowing the playback rate to be adjusted in real-time based on buffer conditions. The player clock can be sped up or slowed down dynamically to match the live stream timeline, transitioning from static local timekeeping to adaptive timekeeping that responds to network conditions.
3Productivity
If encoding timestamps are missing for some media segments, then streaming can continue without interruption, but accurate synchronization requires estimation and correction of missing timestamps
Solution Approach 1:
The patent performs preliminary actions by establishing timestamp correlation relationships between ETs and PTSes early in the playback process. When segments with missing timestamps are encountered, the system can estimate their timestamps based on the established correlation and adjacent segment timestamps, then correct these estimates as more information becomes available.
Solution Approach 2:
The patent uses timestamp copying and interpolation techniques where timestamps from segments with complete information are used to infer timestamps for segments with missing information. The system copies timestamp patterns and intervals from known segments to estimate unknown timestamps, maintaining synchronization without requiring every segment to have explicit timestamp data.
Data Source
AI summary
Techniques for synchronizing playback of multiple media streams are presented, including where presentation timestamps in the multiple media streams may not related. Techniques include determining a first correspondence between the first encoding timestamp and the first presentation time; determining a second correspondence between the second encoding timestamp and the second presentation time; and synchronizing a first player for the first stream and a second player for the second stream based on the first correspondence, the second correspondence, the first and second encoding timestamp, and a playback clock available to both first and second players.