Hierarchical Track Structure for Immersive Media Derivation

Solution Overview

Problem

Conventional storage structures for point cloud content do not support efficient viewport-dependent processing and delivery of 3D immersive media, leading to high computational and bandwidth requirements, as they lack a hierarchical track structure for deriving composite point cloud content from input elementary streams.

Innovation Solution

The implementation of a hierarchical track structure that allows for encoding and decoding of point cloud video data by performing track derivation operations on geometry, attribute, and occupancy data, generating composite immersive media tracks from sub-divided 2D and 3D sub-picture and sub-volumetric tracks, respectively.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If the entire spherical content is processed and delivered, then the user can view the content at any viewport, but the computational and bandwidth requirements become excessively high

Engineering Contradiction:

Improveviewport flexibilityVSAvoidcomputational burden

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The spherical content is divided into multiple viewports or regions, each representing a portion of the 360-degree content. Instead of processing and delivering the entire sphere, the system segments the content into manageable chunks corresponding to different viewing angles, thereby reducing computational and bandwidth requirements while maintaining viewport flexibility.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs partial processing by delivering only the content necessary for the current viewport rather than the complete spherical content. This partial action approach ensures that the user receives sufficient information for their viewing needs without the excessive computational and bandwidth cost of processing the entire sphere.

Inventive Principle:

Principle #16Partial or excessive action

2Adaptability or versatility

If the entire spherical content is processed and delivered, then the user can view the content at any viewport, but the bandwidth consumption increases significantly

Engineering Contradiction:

Improveviewport flexibilityVSAvoidbandwidth consumption

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The spherical content is divided into multiple viewports or regions, each representing a portion of the 360-degree content. Instead of processing and delivering the entire sphere, the system segments the content into manageable chunks corresponding to different viewing angles, thereby reducing computational and bandwidth requirements while maintaining viewport flexibility.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs partial processing by delivering only the content necessary for the current viewport rather than the complete spherical content. This partial action approach ensures that the user receives sufficient information for their viewing needs without the excessive computational and bandwidth cost of processing the entire sphere.

Inventive Principle:

Principle #16Partial or excessive action

3Quantity of substance

If viewport-dependent processing is implemented, then bandwidth and compute resources are saved, but the complexity of content delivery and processing increases

Engineering Contradiction:

Improvebandwidth consumptionVSAvoidprocessing complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by pre-processing and organizing content into predefined viewports or regions before actual playback. This allows the content to be structured in advance, reducing the complexity of real-time viewport-dependent processing by having the content ready in organized segments that can be quickly selected and delivered based on user viewing direction.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11245926B2Methods and apparatus for track derivation for immersive media data tracks

Publication Date: 2022.02.08 MEDIATEK SINGAPORE PTE LTD

AI summary

The techniques described herein relate to methods, apparatus, and computer readable media configured to encode and/or decode video data. Immersive media data is accessed that comprises a hierarchical track structure comprising at least (a) a first track at a first level of the hierarchical track structure comprising first immersive media elementary data, wherein the first track is a parameter track, and the first immersive media elementary data comprises timed metadata, and (b) a second track at a second level in the hierarchical track structure that is different than the first level of the first track, the second track comprising metadata specifying an immersive media track derivation operation. The immersive media track derivation operation is performed on at least the first immersive media elementary data to generate composite immersive media data for the second track.