Immersive 360° Video Communication With Adaptive Viewport Delivery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing mechanisms for immersive media communication, particularly 360° video communication, are inefficient due to high bandwidth requirements and processing power needs, especially in real-time applications, as they do not account for the specific requirements of presenting only a part of the scene based on the user's viewing direction.

Innovation Solution

Extend RTP, RTCP, and SDP protocols to enable efficient immersive media communication by allowing dynamic switching between rendered and non-rendered video data based on latency and user viewing direction, using viewport and viewpoint prediction to optimize data transmission and processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If all immersive media content data is transmitted to the client for rendering, then the client can freely change viewing directions and viewpoints, but the bandwidth requirements and processing power increase significantly

Engineering Contradiction:

Improveviewing direction flexibilityVSAvoidbandwidth requirements

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The immersive media content is segmented into multiple view-dependent representations (e.g., different viewport resolutions, different viewpoint angles). The server transmits only the specific segment corresponding to the client's current viewing direction at full resolution, while other segments are transmitted at lower resolutions or not at all, reducing overall bandwidth while maintaining viewing flexibility.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different regions of the immersive content are assigned different quality levels based on the predicted or current viewing direction. The region corresponding to the user's viewport is transmitted at high quality, while peripheral regions are transmitted at lower quality or compressed more aggressively, optimizing the trade-off between bandwidth and perceived quality.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If all immersive media content data is transmitted to the client for rendering, then the client can freely change viewing directions and viewpoints, but the processing power requirements increase significantly

Engineering Contradiction:

Improveviewing direction flexibilityVSAvoidprocessing power

Core Design Contradiction:

Adaptability or versatilityVSPower

Solution Approach 1:

The server performs preliminary rendering and preprocessing of the immersive content into multiple pre-rendered viewports and viewpoints before transmission. This shifts the computational burden from the client to the server, allowing the client to simply decode and display pre-processed data rather than performing complex real-time rendering calculations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of transmitting the full immersive content and requiring the client to generate all possible views, the server creates and transmits copies of the content from multiple predetermined viewpoints. The client receives these pre-generated view copies and selects the appropriate one for display, significantly reducing client-side processing requirements.

Inventive Principle:

Principle #26Copying

3Ease of operation

If rendered video data is provided to reduce client processing, then latency increases due to additional rendering time at the server

Engineering Contradiction:

Improveclient processing loadVSAvoidlatency

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The system dynamically switches between providing fully rendered video data and providing non-rendered source data based on current network conditions, client capabilities, and latency requirements. When latency is critical, the system provides pre-rendered data with shorter processing pipelines; when bandwidth is sufficient and latency tolerance is higher, it provides source data for client-side rendering.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The server performs preliminary rendering of multiple viewports in advance and stores them ready for rapid transmission. When the client requests content, the server can immediately transmit the pre-rendered viewport data without performing real-time rendering, significantly reducing latency while still offloading processing from the client.

Inventive Principle:

Principle #10Preliminary action

4Productivity

If protocol extensions are implemented to support dynamic switching between rendered and non-rendered data, then the system becomes more complex, but the efficiency of immersive media communication improves

Engineering Contradiction:

Improvecommunication efficiencyVSAvoidprotocol complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The protocol extensions are designed to be multi-functional, supporting not only the switching between rendered and non-rendered data but also handling viewport specification, quality negotiation, and synchronization in a unified framework. This reduces the need for separate protocol mechanisms for each function, managing complexity while enabling multiple features.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12395616B2Immersive media content presentation and interactive 360° video communication

Publication Date: 2025.08.19 FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV

AI summary

An apparatus for presenting immersive media content is described. The apparatus obtains from a sender video data representing immersive content for a certain viewing direction and/or for a certain viewpoint, and displays the video data representing the immersive content for the certain viewing direction and/or for the certain viewpoint.