Mixed NAL Unit Decoding for 360-Degree and ROI Video

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing video coding technologies struggle to properly handle and decode bitstreams containing mixed NAL units of different types, particularly in scenarios involving 360-degree video streaming and Region of Interest (Rol) streaming, where tiles or segments require different random access properties, leading to incorrect reconstruction and playback issues.

Innovation Solution

A decoder is designed to handle bitstreams with mixed NAL unit types by allowing mixing of non-IRAP NAL units such as RASL, RADL, TRAIL, and STSA within access units, and includes signaling mechanisms in parameter sets to manage these mixed types, ensuring correct decoding and reconstruction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If NAL units of different types are mixed in access units to support 360-degree video and Rol streaming, then adaptability and versatility are improved, but device complexity increases due to the need for signaling mechanisms and special handling procedures

Engineering Contradiction:

Improvesupport for 360-degree video and Rol streamingVSAvoiddecoder complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The video stream is divided into different NAL unit types (IDR, CRA, non-RAP) that can be mixed within access units. Each NAL unit type represents a segment with specific random access properties, allowing the decoder to handle different portions of the video stream differently based on the NAL unit type present

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The access unit structure is designed to be universal by allowing it to contain multiple NAL unit types simultaneously. This multi-functional access unit can support both traditional video streaming and emerging applications like 360-degree video and Rol streaming without requiring separate decoding paths

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If mixed NAL unit types are allowed in access units, then random access flexibility is improved, but manufacturing precision deteriorates due to incorrect reconstruction issues

Engineering Contradiction:

Improverandom access flexibilityVSAvoidreconstruction accuracy

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The decoder includes mechanisms to detect the presence of different NAL unit types in access units and adjust its reconstruction process accordingly. This feedback loop ensures that the decoder can identify when mixed NAL unit types are present and apply appropriate handling to maintain reconstruction accuracy

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The decoding process dynamically changes parameters based on the NAL unit types detected in the access unit. When mixed NAL unit types are present, the decoder adjusts its reconstruction parameters to accommodate the different random access properties of each NAL unit type, ensuring accurate reconstruction

Inventive Principle:

Principle #35Parameter changes

3Reliability

If signaling mechanisms are added to manage mixed NAL unit types, then reliability is improved, but device complexity increases

Engineering Contradiction:

Improvedecoding reliabilityVSAvoidsignaling mechanism complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The bitstream includes signaling mechanisms that are prepared in advance to indicate the presence and types of NAL units within access units. This preliminary information is embedded in the bitstream structure itself, allowing the decoder to prepare appropriate handling procedures before actual decoding occurs

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Signaling mechanisms act as an intermediary between the encoded video data and the decoder. These signals provide intermediate information about the NAL unit composition of access units, mediating the decoding process by guiding the decoder on how to handle mixed NAL unit types without requiring complex analysis during decoding

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250294175A1Decoder, encoder and methods for mixing NAL units of different NAL unit types in video streams

Publication Date: 2025.09.18 FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV

AI summary

The present invention is concerned with decoders, encoders and corresponding methods for handling video data streams (11) comprising a first sub-bitstream (11-1) and a second sub-bitstream (11-2). The herein described concept provides solutions for mixing, within an access unit (30, 31, 32), different NAL units (301, 302, 3030, 304) of different NAL unit types. For example, IRAP NAL unit types may be mixed with different IRAP NAL unit types or non-IRAP NAL unit types, and non-IRAP NAL unit types may be mixed with different non-IRAP NAL unit types.