Backward Compatible Immersive Video Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a lack of a format for a backward compatible immersive video stream that can be properly displayed by both legacy and immersive video rendering devices, as legacy devices fail to render immersive videos and vice versa due to differences in encoding and mapping functions.
Innovation Solution
The solution involves encoding a video stream using High Efficiency Video Coding (HEVC) with conformance window parameters carrying the first information and Supplemental Enhancement Information (SEI) carrying the second information, allowing for the creation of a backward compatible frame that can be decoded and rendered by both legacy and immersive devices, using techniques such as equirectangular and cube mapping to adapt the video frame for different display devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a video stream is encoded using immersive mapping functions (equirectangular, cube mapping) for immersive devices, then immersive video quality is improved, but legacy rectangular devices cannot properly display the video due to incompatible encoding formats
Solution Approach 1:
The video stream is segmented into multiple layers: a base layer encoded with rectangular mapping for legacy device compatibility, and enhancement layers encoded with immersive mapping functions for immersive device quality. This segmentation allows each layer to serve its specific device type while maintaining overall compatibility
Solution Approach 2:
A format conversion and adaptation layer is introduced as an intermediary between the immersive video source and the output streams. This mediator converts the immersive content into both rectangular and immersive formats, adding metadata and conformance window parameters that enable legacy devices to properly display adapted versions of the immersive content
2Adaptability or versatility
If a video stream is encoded using rectangular mapping for legacy devices, then legacy device compatibility is improved, but immersive devices cannot properly render the video due to lack of immersive projection information
Solution Approach 1:
The encoding process performs preliminary action by pre-processing the immersive video into multiple formatted streams with embedded metadata before distribution. The rectangular stream includes conformance window parameters and SEI messages that contain immersive mapping information in advance, enabling immersive devices to later reconstruct the full immersive experience without requiring real-time conversion
Solution Approach 2:
The rectangular video stream acts as a nested container that holds both the base rectangular content and embedded metadata about the original immersive mapping. Immersive devices can extract and utilize this nested information to reconstruct the immersive projection, while legacy devices simply use the outer rectangular shell
3Reliability
If separate video streams are created for legacy and immersive devices, then device-specific rendering quality is improved, but system complexity increases due to multiple encoding formats and distribution channels
Solution Approach 1:
A single universal encoding framework is created that produces multi-functional output streams. The same encoding pipeline generates both rectangular and immersive formats with embedded metadata, allowing a single system to serve both legacy and immersive devices without requiring separate encoding infrastructure. The conformance window parameters and SEI messages provide universal compatibility information across device types
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
The present disclosure relates to methods, apparatus or systems for generating, transmitting and decoding a backward compatible immersive video stream. The stream is carrying data representative of an immersive video, composed of a frame organized according to a layout comprising a first area encoded according to a rectangle mapping, a second area encoded according to a mapping transitory from the rectangular mapping to an immersive mapping and a third area encoded according to the immersive mapping. In order to be backward compatible, the stream further comprises a first information representative of the size and the location of the first area within the video frame, and a second information comprising at least the type of the selected layout, the field of view of first part, of the size of said second area within the video frame and a reference direction.