Viewport Marker Sync for Low-Latency Immersive Video Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently delivering high-quality, viewport-dependent 360-degree video content with minimal latency and synchronization issues, particularly in WebRTC-based browser viewers, due to the lack of in-band techniques for viewport orientation data and frame synchronization.
Innovation Solution
A method and apparatus that encode viewport orientation data as visual markers within the video frames, using a color code or watermark, allowing synchronized delivery without requiring custom media transport payloads, and employing a ring buffer to manage viewport updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If viewport orientation data and frame synchronization are delivered using custom media transport payloads, then synchronization precision is improved, but device complexity increases due to lack of browser support
Solution Approach 1:
The patent uses visual markers as an intermediary carrier to transport viewport orientation data and frame synchronization information. Instead of relying on complex custom media transport payloads, the markers serve as a universal intermediary that can be processed by standard browser video playback mechanisms, thereby achieving precise synchronization without increasing device complexity
Solution Approach 2:
The patent replaces the mechanical system of custom media transport payloads with an optical/visual system. By encoding synchronization data in visual markers within video frames, the system leverages the browser's existing video decoding capabilities rather than requiring specialized transport layer support, thus simplifying the implementation while maintaining precision
2Manufacturing precision
If high-quality viewport-dependent 360-degree video content is delivered, then manufacturing precision is improved, but loss of time increases due to latency
Solution Approach 1:
The patent segments the 360-degree omnidirectional video into multiple viewports corresponding to different head orientations. By processing and transmitting only the relevant viewport segments rather than the entire sphere, the system reduces data volume and processing time, thereby decreasing motion-to-photon latency while maintaining rendering precision for the active viewing area
Solution Approach 2:
The system performs preliminary actions by pre-calculating and pre-encoding viewport regions based on predicted head orientation or by buffering synchronized marker-data pairs in advance. This allows the playback system to quickly retrieve and render the correct viewport without waiting for real-time processing, thus reducing latency while maintaining precision
3Ease of operation
If viewport orientation data is encoded as visual markers within video frames, then ease of operation is improved through synchronized delivery, but loss of information increases due to marker encoding overhead
Solution Approach 1:
The patent utilizes color changes or color-coded markers within the video frames to encode viewport orientation data. By leveraging the existing color information processing capabilities of video decoders, the system achieves easy synchronized operation without requiring separate data streams. The color-coded markers provide compact encoding that minimizes information loss compared to alternative text-based or metadata approaches
Data Source
AI summary
The embodiments relate to a method and technical equipment for implementing the method. The method comprises receiving an omnidirectional video; determining a marker identifying a viewport for a client device; determining a region of the omnidirectional video frame based, at least partially, on the viewport; encoding the determined region with a marker for synchronized operation between the client device and a server; packetizing the encoded region; and transmitting the packetized region to the client device.