Video encoding method, video encoding apparatus, video decoding method, video decoding apparatus, method for transmitting bitstream, and recording medium in which bitstream is stored

Abstract

A method according to embodiments comprises the steps of: acquiring a digitally signed content initialization SEI message from a bitstream; acquiring a digitally signed content selection SEI message from the bitstream; acquiring, from the bitstream, a digitally signed content verification SEI message; and decoding a picture in the bitstream. A method according to embodiments comprises the steps of: encoding a picture; generating a digitally signed content initialization SEI message; generating a digitally signed content selection SEI message; and generating a digitally signed content verification SEI message.

Description

Image encoding method, image encoding device, image decoding method, image decoding device, method for transmitting a bitstream and a recording medium storing a bitstream

[0001] The embodiments relate to an image encoding method, an image encoding device, an image decoding method, an image decoding device, a method for transmitting a bitstream, and a recording medium storing a bitstream.

[0002] Recently, the demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition), has been increasing across various fields. As video data becomes higher in resolution and quality, the relative amount of information or bits transmitted increases compared to conventional video data. This increase in the amount of transmitted information or bits leads to higher transmission and storage costs.

[0003] Accordingly, high-efficiency video compression technology is required to effectively transmit, store, and play back high-resolution, high-quality video information.

[0004] The embodiments provide an image encoding method, an image encoding device, an image decoding method, an image decoding device, a method for transmitting a bitstream, and a recording medium storing a bitstream.

[0005] The embodiments provide an image encoding method with improved encoding and decoding efficiency, an image encoding device, an image decoding method, an image decoding device, a method for transmitting a bitstream, and a recording medium storing a bitstream.

[0006] However, the scope of rights of the embodiments is not limited to the technical problems described above, and may be extended to other technical problems that a person skilled in the art can infer based on the entire content described.

[0007] A method according to embodiments comprises the steps of: obtaining a digitally signed content initialization SEI message from a bitstream; obtaining a digitally signed content selection SEI message from a bitstream; obtaining a digitally signed content verification SEI message from a bitstream; and decoding a picture within a bitstream. A method according to embodiments comprises the steps of: encoding a picture; generating a digitally signed content initialization SEI message; generating a digitally signed content selection SEI message; and generating a digitally signed content verification SEI message.

[0008] The embodiments provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0009] The embodiments provide a non-transient computer-readable recording medium that stores a bitstream generated by an image encoding method.

[0010] The embodiments provide a non-transient computer-readable recording medium that stores a bitstream received and decoded by an image decoding device and used for image restoration.

[0011] The embodiments provide a method for transmitting a bitstream generated by an image encoding method.

[0012] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0013] Drawings are included to further understand the embodiments, and the drawings illustrate the embodiments along with descriptions related to the embodiments. For a better understanding of the various embodiments described below, one must refer to the description of the embodiments below in relation to the following drawings, which include parts corresponding to similar reference numerals throughout the drawings.

[0014] FIG. 1 shows a video and / or image coding system according to embodiments.

[0015] FIG. 2 shows an encoding device according to embodiments.

[0016] FIG. 3 shows a decoding device according to embodiments.

[0017] Figure 4 shows the structure of a content streaming system according to embodiments.

[0018] FIG. 5 shows an example of a picture divided into Coding Tree Units (CTUs) according to embodiments.

[0019] FIG. 6 shows an example of a picture partitioned into tiles and raster-scan slices according to embodiments.

[0020] FIG. 7 shows an example of a picture partitioned into tiles and raster-scan slices according to embodiments.

[0021] FIG. 8 shows an example of a picture partitioned into tiles, bricks, and rectangular slices according to embodiments.

[0022] FIG. 9 shows an example of a picture including subpictures according to embodiments.

[0023] FIG. 10 shows an example of a picture including tiles and CTUs according to embodiments.

[0024] FIG. 11 shows a multi-type tree splitting mode according to embodiments.

[0025] FIG. 12 shows splitting flags within a quad tree of a multi-type tree coding structure according to embodiments.

[0026] FIG. 13 shows an example of a quad tree of a multi-type tree coding block structure according to embodiments.

[0027] FIG. 14 shows the prohibition of TT (Ternary Tree) division for coding blocks according to embodiments.

[0028] FIG. 15 shows the transform and inverse transform according to the embodiments.

[0029] FIG. 16 shows a Low-Frequency Non-Separable Transform (LFNST) according to embodiments.

[0030] FIG. 17 shows CABAC (Context Adaptive Binary Arithmetic Coding) encoding according to embodiments.

[0031] FIG. 18 illustrates an entropy encoding method according to embodiments.

[0032] FIG. 19 illustrates an entropy decoding method according to embodiments.

[0033] FIG. 20 illustrates a picture decoding method according to embodiments.

[0034] FIG. 21 illustrates a picture encoding method according to embodiments.

[0035] FIG. 22 shows a hierarchical structure for a coded image according to embodiments.

[0036] FIGS. 23a, FIGS. 23b, FIGS. 23c, FIGS. 23d, and FIGS. 23e show picture header structures according to embodiments.

[0037] FIG. 24 shows the digitally signed content initialization SEI message syntax according to the embodiments.

[0038] FIG. 25 shows a digitally signed content selection SEI message syntax according to embodiments.

[0039] FIG. 26 shows the digitally signed content verification SEI message syntax according to the embodiments.

[0040] FIG. 27 shows the current structure of RefDigest according to the embodiments.

[0041] FIG. 28 illustrates a encoding method according to embodiments.

[0042] FIG. 29 illustrates a decoding method according to embodiments.

[0043] Preferred embodiments of the embodiments are described in detail, and examples thereof are shown in the accompanying drawings. The detailed description below, with reference to the accompanying drawings, is intended to describe preferred embodiments of the embodiments rather than merely embodiments that may be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments can be practiced without these details.

[0044] Most terms used in the embodiments are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, the embodiments should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0045] Related technical fields: Versatile Video Coding (VVC), Versatile supplemental enhancement information messages for coded video bitstreams (VSEI), Additional SEI messages for VSEI (Draft 3), SEI processing order and processing order nesting SEI messages in VVC (draft 7), Technologies under consideration for future extensions of VSEI (draft 4), SEI messages for VSEI version 4 (Draft 2).

[0046]

[0047] FIG. 1 shows a video and / or image coding system according to embodiments.

[0048] As shown in FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in the form of a file or streaming via a digital storage medium or a network.

[0049] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be called a video / image encoding device, and the decoding device may be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be composed of a separate device or an external component.

[0050] A video source may acquire video / images through processes such as video / image capture, synthesis, or generation. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, etc., and may generate video / images (electronically). For example, virtual video / images may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process in which related data is generated.

[0051] The encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0052] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device in the form of a file or streaming via a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0053] The decoding device can decode video / images by performing a series of procedures such as inverse quantization, inverse transform, and prediction corresponding to the operation of the encoding device.

[0054] The renderer can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0055] This document relates to video / video coding. For example, the methods / executions disclosed in this document may be applied to methods disclosed in the VVC (versatile video coding) standard, EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard) or next-generation video / video coding standards (e.g., H.267 or H.268).

[0056] This document presents various embodiments regarding video / image coding, and unless otherwise noted, the embodiments may be performed in combination with one another.

[0057] In this document, "video" may refer to a set of images over time. "Picture" generally refers to a unit representing a single image at a specific time, and "slice" or "tile" are units that constitute a part of a picture in coding. A slice or tile may contain one or more CTUs (coding tree units). A single picture may consist of one or more slices or tiles. A single picture may consist of one or more tile groups. A tile group may contain one or more tiles. A "brick" may represent a rectangular area of ​​rows of CTUs within a tile in a picture.

[0058] A brick can represent a rectangular area of ​​a row of CTUs within a tile in a picture. A tile can be divided into multiple bricks, and each brick consists of one or more rows of CTUs within the tile. A tile that is not divided into multiple bricks is also referred to as a brick. A brick scan is a specific sequential order of CTUs that divides a picture. In a brick, CTUs are arranged sequentially as CTU raster scans; in a brick within a tile, bricks are arranged sequentially as brick raster scans of the tile; and in a tile within a picture, tiles are arranged sequentially as tile raster scans of the picture. A tile is a rectangular area of ​​CTUs within a specific tile column and a specific tile row in a picture. A tile column is a rectangular area of ​​CTUs that is equal to the height of the picture and has a width specified by the syntax element of the picture parameter set. A tile row is a rectangular area of ​​CTUs that has a height specified by the syntax element of the picture parameter set and a width equal to the width of the picture. A tile scan is a specific sequential order of CTUs that divides a picture. In tiles, CTUs are continuously aligned by CTU raster scans, and in pictures, tiles are continuously aligned by tile raster scans. A slice contains an integer number of bricks of a picture, which can be contained exclusively in a single NAL unit. A slice can consist of multiple complete tiles or a sequence in which the complete bricks of a single tile are arranged continuously.

[0059] In this document, tile group and slice may be used interchangeably. For example, in this document, tile group / tile group header may be referred to as slice / slice header.

[0060] A pixel or pel can refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. Generally, a sample can represent a pixel or its value, and it may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component.

[0061] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. A unit may include one luminance block and two chroma (e.g., cb, cr) blocks. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0062] FIG. 2 shows an encoding device according to embodiments.

[0063] FIG. 2 shows a schematic block diagram of an encoding device to which the embodiment(s) of the present document can be applied and to which video / image signal encoding is performed.

[0064] As shown in FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-predictor (221) and an intra-predictor (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit (250) may be referred to as a reconstructor or a reconstructed block generator. The above-described image segmentation unit (210), prediction unit (220), residual processing unit (230), entropy encoding unit (240), addition unit (250), and filtering unit (260) may be configured by one or more hardware components (e.g., an encoder chipset or processor) according to the embodiment. Additionally, the memory (270) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0065] The image segmentation unit (210) can divide an input image (or picture, frame) input to an encoding device (200) into one or more processing units. For example, a processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, a single coding unit may be divided into multiple coding units of a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first and the binary-tree structure and / or ternary structure may be applied later. Or, the binary-tree structure may be applied first. A coding procedure according to this document may be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively divided into lower-depth coding units so that a coding unit of the optimal size is used as the final coding unit. Here, the term "coding procedure" may include procedures such as prediction, transformation, and restoration described below. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the aforementioned final coding unit. The prediction unit may be a unit for sample prediction, and the transformation unit may be a unit for deriving transformation coefficients and / or a unit for deriving a residual signal from transformation coefficients.

[0066] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.

[0067] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, as illustrated, the unit that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) within the encoder (200) may be called a subtraction unit (231). The prediction unit performs a prediction for a block to be processed (hereinafter referred to as the current block) and can generate a predicted block containing prediction samples for the current block. The prediction unit can determine whether intra prediction is applied or inter prediction is applied at the current block or CU level. The prediction unit can generate various information regarding prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (240). The information regarding prediction can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0068] The intra prediction unit (222) can predict the current block by referencing samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a Planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0069] The inter prediction unit (221) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference picture indices. Motion information may further include information on inter prediction directions (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different. Temporal surrounding blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc., and a reference picture containing temporal surrounding blocks may be referred to as a collocated picture (colPic). For example, the inter prediction unit (221) may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of surrounding blocks as motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0070] The prediction unit (220) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. IBC may utilize at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within the picture can be signaled based on information regarding the palette table and palette index.

[0071] The prediction signal generated through the prediction unit (including the inter prediction unit (221) and / or the intra prediction unit (222)) may be used to generate a restored signal or to generate a residual signal. The transformation unit (232) may generate transform coefficients by applying a transformation technique to the residual signal. For example, the transformation technique may include at least one of the Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loeve Transform (KLT), Graph-Based Transform (GBT), or Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transformation obtained based on generating a prediction signal using all previously reconstructed pixels. In addition, the transformation process can be applied to pixel blocks of the same square size, or to non-square blocks of variable size.

[0072] The quantization unit (233) quantizes the transformation coefficients and transmits them to the entropy encoding unit (240), and the entropy encoding unit (240) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (233) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients. The entropy encoding unit (240) can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (240) may encode information necessary for video / image restoration (e.g., values ​​of syntax elements) together or separately, in addition to the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information regarding various parameter sets, such as an Adaptation Parameter Set (APS), a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), or a Video Parameter Set (VPS). Additionally, the video / image information may further include general constraint information. In this document, information and / or syntax elements transmitted / signaled from the encoding device to the decoding device may be included in the video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream.The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits the signal output from the entropy encoding unit (240) and / or a storage unit (not shown) that stores it may be configured as internal / external elements of the encoding device (200), or the transmission unit may be included in the entropy encoding unit (240).

[0073] Quantized transformation coefficients output from the quantization unit (233) can be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit (234) and the inverse transformation unit (235). An adder (155) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the inter-prediction unit (221) or the intra-prediction unit (222). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (250) may be called a reconstruction unit or a reconstruction block generation unit. The generated restoration signal can be used for intra prediction of the next processing target block within the current picture, and can also be used for inter prediction of the next picture after filtering as described below.

[0074] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0075] The filtering unit (260) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (260) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (270), specifically in the DPB of memory (270). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240), as described below in the description of each filtering method. The information regarding filtering can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0076] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter-prediction unit (221). Through this, when inter-prediction is applied, the encoding device can avoid prediction mismatches between the encoding device (100) and the decoding device, and can also improve encoding efficiency.

[0077] The memory (270) DPB can store the modified restored picture to be used as a reference picture in the inter-prediction unit (221). The memory (270) can store motion information of blocks from which motion information is derived (or encoded) within the current picture and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (221) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (270) can store restoration samples of the blocks restored within the current picture and transmit them to the intra-prediction unit (222).

[0078] FIG. 3 shows a decoding device according to embodiments.

[0079] FIG. 3 shows a schematic block diagram of a decoding device to which the embodiment(s) of the present document can be applied and to which decoding of a video / image signal is performed.

[0080]

[0081] As shown in FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-predictor (331) and an intra-predictor (332). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). The aforementioned entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) may be configured by a single hardware component (e.g., a decoder chipset or a processor) according to an embodiment. Additionally, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0082] When a bitstream containing video / image information is input, the decoding device (300) can restore the image in correspondence with the process in which the video / image information is processed by the encoding device of FIG. 2. For example, the decoding device (300) can derive units / blocks based on block division information obtained from the bitstream. The decoding device (300) can perform decoding using a processing unit applied by the encoding device. Thus, the processing unit for decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a binary tree structure. One or more conversion units may be derived from the coding unit. And, the restored image signal decoded and output through the decoding device (300) can be played back through a playback device.

[0083] The decoding device (300) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through the entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The decoding device can decode the picture based further on information regarding the parameter sets and / or general constraint information. The signaling / receiving information and / or syntax elements described below in this document can be obtained from the bitstream by decoding through a decoding procedure. For example, the entropy decoding unit (310) can decode information within a bitstream based on coding methods such as exponential chord coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in the bitstream, determine a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of a symbol / bin decoded in the previous step, predict the probability of occurrence of the bin according to the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using information of the decoded symbol / bin for the context model of the next symbol / bin after determining the context model.Information regarding prediction among the information decoded in the entropy decoding unit (310) is provided to the prediction unit (inter prediction unit (332) and intra prediction unit (331)), and residual values ​​for which entropy decoding has been performed in the entropy decoding unit (310), for example, quantized transformation coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive residual signals (residual blocks, residual samples, residual sample arrays). Additionally, information regarding filtering among the information decoded in the entropy decoding unit (310) can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of the entropy decoding unit (310). Meanwhile, the decoding device according to the present document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include an entropy decoding unit (310), and the sample decoder may include at least one of an inverse quantization unit (321), an inverse transform unit (322), an adder (340), a filtering unit (350), a memory (360), an inter prediction unit (332), and an intra prediction unit (331).

[0084] In the inverse quantization unit (321), the quantized transformation coefficients can be inversely quantized to output transformation coefficients. The inverse quantization unit (321) can rearrange the quantized transformation coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit (321) can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain transformation coefficients.

[0085] In the inverse conversion unit (322), the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).

[0086] The prediction unit can perform a prediction for the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit (310), the prediction unit can determine whether an intra prediction or an inter prediction is applied to the current block and can determine a specific intra / inter prediction mode.

[0087] The prediction unit (320) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. IBC may utilize at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, information regarding the palette table and palette index may be included in the video / image information and signaled.

[0088] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit (331) may determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0089] The inter prediction unit (332) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. Motion information may include a motion vector and a reference picture index. Motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) may construct a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and information regarding the prediction may include information indicating the mode of inter prediction for the current block.

[0090] The adder (340) can generate a restoration signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the prediction signal (predicted block, predicted sample array) output from the prediction unit (including the inter prediction unit (332) and / or the intra prediction unit (331)). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the restoration block.

[0091] The addition unit (340) may be called a restoration unit or a restoration block generation unit. The generated restoration signal may be used for intra-predicting the next block to be processed within the current picture, may be output after filtering as described below, or may be used for inter-predicting the next picture.

[0092] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0093] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can generate a modified restored picture by applying various filtering methods to the restored picture, and can transmit the modified restored picture to memory (360), specifically to the DPB of memory (360). Various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0094] The (modified) restored picture stored in the DPB of the memory (360) can be used as a reference picture in the inter-prediction unit (332). The memory (360) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (260) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (360) can store restoration samples of blocks restored within the current picture and transmit them to the intra-prediction unit (331).

[0095] In this specification, the embodiments described in the filtering unit (260), inter prediction unit (221), and intra prediction unit (222) of the encoding device (100) may be applied to the filtering unit (350), inter prediction unit (332), and intra prediction unit (331) of the decoding device (300) in the same or corresponding manner.

[0096] Implementation and Application Examples:

[0097] The embodiments described in this document may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information on instructions or algorithms for implementation may be stored on a digital storage medium.

[0098] In addition, the decoding device and encoding device to which the embodiment(s) of this document apply may be included in multimedia broadcasting transmission and reception devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video conversation devices, real-time communication devices such as video communication, mobile streaming devices, storage media, camcorders, Video on Demand (VoD) service providers, Over-the-top video (OTT) devices, internet streaming service providers, 3D video devices, virtual reality (VR) devices, augmented reality (AR) devices, video phone video devices, transportation terminals (e.g., vehicle terminals (including autonomous vehicles), airplane terminals, ship terminals, etc.), and medical video devices, and may be used to process video signals or data signals. For example, Over-the-top video (OTT) devices may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, Digital Video Recorders (DVRs), etc.

[0099] Additionally, the processing method to which the embodiment(s) of this document are applied may be produced in the form of a program that is executed by a computer and may be stored on a computer-readable recording medium. Multimedia data having a data structure according to the embodiment(s) of this document may also be stored on a computer-readable recording medium. A computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. A computer-readable recording medium may include, for example, a Blu-ray disc (BD), a Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, a floppy disk, and an optical data storage device. Additionally, a computer-readable recording medium includes media implemented in the form of a carrier wave (e.g., transmission over the Internet). Additionally, a bitstream generated by an encoding method may be stored on a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0100] Additionally, the embodiment(s) of this document may be implemented as a computer program product by program code, and the program code may be executed on a computer by the embodiment(s) of this document. The program code may be stored on a computer-readable carrier.

[0101] Figure 4 shows the structure of a content streaming system according to embodiments.

[0102] A content streaming system to which the embodiment(s) of this document apply may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0103] The encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data to generate a bitstream, and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server can be omitted.

[0104] A bitstream may be generated by an encoding method or a bitstream generation method to which the embodiment(s) of this document are applied, and a streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0105] The streaming server transmits multimedia data to the user's device based on user requests made through the web server, while the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server forwards the request to the streaming server, which then transmits the multimedia data to the user. In this process, the content streaming system may include a separate control server, which plays the role of managing commands and responses between devices within the content streaming system.

[0106] A streaming server can receive content from a media storage and / or an encoding server. For example, if content is received from an encoding server, it can be received in real time. In this case, to provide a seamless streaming service, the streaming server may store the bitstream for a certain period of time.

[0107] Examples of user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0108] Each server within the content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0109] Partitioning structure:

[0110] The video / image coding method according to this document can be performed based on the following partitioning structure. Specifically, the procedures described below, such as prediction, residual processing ((inverse)transform, (inverse)quantization, etc.), syntax element coding, and filtering, can be performed based on CTU and CU (and / or TU, PU) derived based on the partitioning structure. The block partitioning procedure is performed in the image splitting unit (210) of the encoding device described above, and the partitioning-related information can be processed (encoded) in the entropy encoding unit (240) and transmitted to the decoding device in the form of a bitstream. The entropy decoding unit (310) of the decoding device can derive the block partitioning structure of the current picture based on the partitioning-related information obtained from the bitstream, and perform a series of procedures for image decoding (e.g., prediction, residual processing, block / picture restoration, in-loop filtering, etc.) based thereon. The CU size and the TU size may be the same, or multiple TUs may exist within the CU area. Meanwhile, the term CU size generally refers to the luminance component (sample) CB size. The term TU size generally refers to the luminance component (sample) TB size. The chroma component (sample) CB or TB size can be derived based on the luminance component (sample) CB or TB size according to the component ratio based on the color format (chroma format, e.g., 4:4:4, 4:2:2, 4:2:0, etc.) of the picture / image. The TU size can be derived based on maxTbSize. For example, if the CU size is greater than maxTbSize, multiple TUs (TBs) of maxTbSize are derived from C, and conversion / inverse conversion can be performed in TU (TB) units. In addition, for example, when intra prediction is applied, the intra prediction mode / type is derived in units of CU (or CB), and the procedure for deriving surrounding reference samples and generating prediction samples can be performed in units of TU (or TB).In this case, one or more TUs (or TBs) may exist within a single CU (or CB) region, and in this case, the multiple TUs (or TBs) may share the same intra prediction mode / type.

[0111] Additionally, in the coding of video / images according to this document, image processing units may have a hierarchical structure. A picture may be divided into one or more tiles, bricks, slices, and / or tile groups. A slice may contain one or more bricks. A brick may contain one or more CTU rows within a tile. A slice may contain an integer number of bricks in a picture. A tile group may contain one or more tiles. A tile may contain one or more CTUs. A CTU may be divided into one or more CUs. A tile is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. A tile group may contain an integer number of tiles based on a tile raster scan within a picture. A slice header may carry information / parameters that can be applied to the corresponding slice (blocks within the slice). If the encoding / decoding device has a multi-core processor, the encoding / decoding procedures for tiles, slices, bricks, and / or tile groups may be processed in parallel. In this document, the terms slice and tile group may be used interchangeably. A tile group header may be referred to as a slice header. Here, a slice may have one of the slice types, including intra (I) slice, predictive (P) slice, and bi-predictive (B) slice. For blocks within an I slice, inter-prediction is not used for prediction, and only intra-prediction may be used. Of course, even in this case, the original sample value may be coded and signaled without prediction.For blocks within a P slice, intra prediction or inter prediction may be used, and if inter prediction is used, only uni prediction may be used. Meanwhile, for blocks within a B slice, intra prediction or inter prediction may be used, and if inter prediction is used, up to bi prediction may be used.

[0112] In the encoder, tile / tile group, brick, slice, and maximum and minimum coding unit sizes are determined based on video characteristics (e.g., resolution) or by considering coding efficiency or parallel processing, and information regarding this or information that can derive it may be included in the bitstream.

[0113] The decoder can obtain information indicating whether the tile / tile group, brick, slias, and CTU within the tile of the current picture have been divided into multiple coding units. Efficiency can be increased by obtaining (transmitting) this information only under specific conditions.

[0114] A slice header (slice header syntax) may include information / parameters that can be applied commonly to slices. An APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be applied commonly to one or more pictures. An SPS (SPS syntax) may include information / parameters that can be applied commonly to one or more sequences. A VPS (VPS syntax) may include information / parameters that can be applied commonly to multiple layers. A DPS (DPS syntax) may include information / parameters that can be applied commonly across the video. A DPS may include information / parameters related to the concatenation of a CVS (coded video sequence).

[0115] In this document, the term "higher-level syntax" may include at least one of APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, and slice header syntax.

[0116] In addition, for example, information regarding the division and configuration of tiles / tile groups / bricks / slices can be configured at the encoding stage through high-level syntax and transmitted to a decoding device in the form of a bitstream.

[0117] FIG. 5 shows an example of a picture divided into Coding Tree Units (CTUs) according to embodiments.

[0118] Partitioning of picture into CTUs:

[0119] Pictures can be divided into a sequence of coding tree units (CTUs). A CTU may correspond to a coding tree block (CTB). Alternatively, a CTU may include a coding tree block of luminance samples and two coding tree blocks of corresponding chroma samples. In other words, for a picture containing three sample arrays, a CTU may include an NxN block of luminance samples and two corresponding blocks of chroma samples. FIG. 5 illustrates an example in which a picture is divided into CTUs.

[0120] The maximum allowable size of a CTU for coding and prediction, etc., may differ from the maximum allowable size of a CTU for transformation. For example, the maximum allowable size of a luminance block within a CTU may be 128x128 (even though the maximum size of luminance ring blocks is 64x64).

[0121] FIG. 6 shows an example of a picture partitioned into tiles and raster-scan slices according to embodiments.

[0122] Partitioning of pictures into subpictures, slices, and tiles:

[0123] A picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of CTUs that includes a rectangular area of ​​the picture. The CTUs within a tile are scanned in the raster scan order within that tile.

[0124] A slice consists of an integer number of complete tiles or an integer number of consecutive complete CTU rows within a picture tile.

[0125] Two modes are supported for slicing: raster scan slice mode and rectangular slice mode. In raster scan slice mode, a slice contains a complete sequence of tiles from a tile raster scan of the picture. In rectangular slice mode, a slice contains multiple complete tiles that make up a rectangular area of ​​the picture, or multiple consecutive rows of complete CTUs of a single tile that make up a rectangular area of ​​the picture. The tiles within a rectangular slice are scanned in the tile raster scan order within the rectangular area corresponding to that slice.

[0126] A sub-picture consists of one or more slices that cover the entire rectangular area of ​​the picture.

[0127] Figure 6 shows an example of splitting a picture into raster scan slices. Here, the picture is divided into 12 tiles and 3 raster scan slices.

[0128] FIG. 7 shows an example of a picture partitioned into tiles and raster-scan slices according to embodiments.

[0129] Figure 7 shows an example of dividing a picture into rectangular slices. Here, the picture is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular slices.

[0130] FIG. 8 shows an example of a picture partitioned into tiles, bricks, and rectangular slices according to embodiments.

[0131] Figure 8 shows an example of a picture divided into tiles and rectangular slices. Here, the picture is divided into 4 tiles (2 tile columns and 2 tile rows) and 4 rectangular slices.

[0132] FIG. 9 shows an example of a picture including subpictures according to embodiments.

[0133] Fig. 9 shows an example of sub-picture division of a picture. Here, the picture is divided into 28 sub-pictures of various dimensions.

[0134] FIG. 10 shows an example of a picture including tiles and CTUs according to embodiments.

[0135] If the picture is coded using three separate color planes (where separate_colour_plane_flag is 1), the slice contains only one CTU of a color component identified by its color_plane_id value, and each array of color components in the picture consists of slices having the same color_plane_id value. Coded slices with different color_plane_id values ​​within the picture may be interleaved with each other under the constraint that for each color_plane_id value, the coded slice NAL unit having that color_plane_id value must be in ascending order of CTU addresses in the tile scan order for the first CTU of each coded slice NAL unit.

[0136] Note - If separate_colour_plane_flag is 0, each CTU of the picture is contained in exactly one slice. If separate_colour_plane_flag is 1, each CTU of the color component is contained in exactly one slice (information for each CTU of the picture exists in exactly three slices, and these three slices have different colour_plane_id values).

[0137] Tile changes the order of CTUs in a picture. If the picture is divided into two or more tiles, the order of CTUs is the raster scan order within each tile, as shown in FIG. 10. In FIG. 10, the picture is divided into two tiles, and each tile has eight CTUs. The order of CTUs within the tile is the raster scan order.

[0138] FIG. 11 shows a multi-type tree splitting mode according to embodiments.

[0139] Partitioning of the CTUs using a tree structure

[0140] A CTU can be partitioned into CUs based on a quad-tree (QT) structure. The quad-tree structure can be referred to as a quaternary tree structure. This is intended to reflect various local characteristics. Meanwhile, in this document, a CTU can be partitioned based on a multitype tree structure partitioning that includes not only quad-trees but also binary trees (BT) and ternary trees (TT). Hereinafter, the term QTBT structure may include quad-tree and binary tree-based partitioning structures, and QTBTTT may include quad-tree, binary tree, and ternary tree-based partitioning structures. Alternatively, the QTBT structure may include quad-tree, binary tree, and ternary tree-based partitioning structures. In a coding tree structure, CUs can have a square or rectangular shape. A CTU can first be partitioned into a quad-tree structure. Subsequently, the leaf nodes of the quad-tree structure can be further partitioned by a multitype tree structure. For example, as shown in FIG. 11, a multitype tree structure may include four partition types in a schematic manner.

[0141] The four splitting types may include vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER), and horizontal ternary splitting (SPLIT_TT_HOR). Leaf nodes of a multitype tree structure may be called CUs. These CUs can be used for prediction and transformation procedures. In this document, CUs, PUs, and TUs generally have the same block size. However, if the maximum supported transform length is smaller than the width or height of the color component of the CU, the CU and TU may have different block sizes.

[0142] FIG. 12 shows splitting flags within a quad tree of a multi-type tree coding structure according to embodiments.

[0143] FIG. 12 exemplarily illustrates the signaling mechanism of partition splitting information in a quadtree with nested multi-type tree structure.

[0144] Here, the CTU is treated as the root of the quadtree and is initially partitioned into a quadtree structure. Each quadtree leaf node can subsequently be further partitioned into a multitype tree structure. In the multitype tree structure, a first flag (e.g., mtt_split_cu_flag) is signaled to indicate whether the node is further partitioned. If the node is further partitioned, a second flag (e.g., mtt_split_cu_vertical_flag) may be signaled to indicate the splitting direction. Subsequently, a third flag (e.g., mtt_split_cu_binary_flag) may be signaled to indicate whether the splitting type is binary or binary. For example, based on mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree splitting mode (MttSplitMode) of CU can be derived as shown in Table 1 (MttSplitMode derviation based on multi-type tree syntax elements).

[0145] [Table 1]

[0146]

[0147] FIG. 13 shows an example of a quad tree of a multi-type tree coding block structure according to embodiments.

[0148] FIG. 13 exemplarily illustrates a CTU being divided into multiple CUs based on a quadtree and nested multi-type tree structure.

[0149] Here, bold block edges represent quadtree partitioning, and the remaining edges represent multitype tree partitioning. Quadtree partitioning involving a multitype tree can provide a content-adapted coding tree structure. A CU can correspond to a coding block (CB). Alternatively, a CU may include a coding block of luminance samples and two coding blocks of corresponding chroma samples. The size of a CU may be as large as a CTU, or it may be 4x4 in luminance sample units. For example, in the case of a 4:2:0 color format (or chroma format), the maximum chroma CB size may be 64x64 and the minimum chroma CB size may be 2x2.

[0150] For example, in this document, the maximum allowable luma TB size may be 64x64 and the maximum allowable chroma TB size may be 32x32. If the width or height of a CB partitioned according to the tree structure is greater than the maximum conversion width or height, the CB may be automatically (or implicitly) partitioned until the horizontal and vertical TB size limits are satisfied.

[0151] Meanwhile, for a quadtree coding tree scheme involving a multitype tree, the following parameters can be defined and identified as SPS syntax elements.

[0152] CTU size: Size of the root node of a 4th-order tree

[0153] MinQTSize: Minimum allowed 4th-order tree leaf node size

[0154] MaxBtSize: Maximum allowed binary tree root node size

[0155] MaxTtSize: Maximum allowed ternary tree root node size

[0156] MaxMttDepth: The maximum allowed hierarchy depth of a multi-type tree splitting at a 4th-order tree leaf.

[0157] MinBtSize: Minimum allowed binary tree leaf node size

[0158] MinTtSize: Minimum allowed tertiary tree leaf node size

[0159] As an example of a quadtree coding tree structure involving a multitype tree, the CTU size can be set to 64x64 blocks of 128x128 luminance samples and two corresponding chroma samples (in the 4:2:0 chroma format). In this case, MinOTSize can be set to 16x16, MaxBtSize to 128x128, MaxTtSize to 64x64, MinBtSize and MinTtSize (for both width and height) to 4x4, and MaxMttDepth to 4. Quadtree partitioning can be applied to the CTU to create quadtree leaf nodes. Quadtree leaf nodes can be called leaf QT nodes. Quadtree leaf nodes can have sizes ranging from 16x16 (i.e., the MinOTSize) to 128x128 (i.e., the CTU size). If the leaf QT node is 128x128, it may not be further split into a binary tree / binary tree. This is because even if it were split in this case, it would exceed MaxBtsize and MaxTtsize (i.e., 64x64). Otherwise, the leaf QT node may be further split into a multitype tree. Therefore, the leaf QT node is the root node of the multitype tree, and the leaf QT node can have a multitype tree depth (mttDepth) value of 0. If the multitype tree depth reaches MaxMttdepth (e.g., 4), further splitting may not be considered. If the width of the multitype tree node is equal to MinBtSize and is less than or equal to 2xMinTtSize, further horizontal splitting may not be considered. If the height of a multitype tree node is equal to MinBtSize and less than or equal to 2xMinTtSize, no further vertical splitting may be considered.

[0160] FIG. 14 shows the prohibition of TT (Ternary Tree) division for coding blocks according to embodiments.

[0161] In order to allow 64x64 luminance block and 32x32 chroma pipeline designs in a hardware decoder, TT splitting may be forbidden in certain cases. For example, if the width or height of the luminance coding block is greater than 64, TT splitting may be forbidden, as shown in FIG. 14. Also, for example, if the width or height of the chroma coding block is greater than 32, TT splitting may be forbidden.

[0162] In this document, the coding tree scheme may support Luma and Chroma (component) blocks having separate block tree structures. If Luma and Chroma blocks within a single CTU have the same block tree structure, it may be denoted as SINGLE_TREE. If Luma and Chroma blocks within a single CTU have separate block tree structures, it may be denoted as DUAL_TREE. In this case, the block tree type for the Luma component may be called DUAL_TREE_LUMA, and the block tree type for the Chroma component may be called DUAL_TREE_CHROMA. For P and B slice / tile groups, Luma and Chroma CTBs within a single CTU may be restricted to having the same coding tree structure. However, for I slice / tile groups, Luma and Chroma blocks may have separate block tree structures. If individual block tree mode is applied, the Luma CTB may be divided into CUs based on a specific coding tree structure, and the Chroma CTB may be divided into Chroma CUs based on a different coding tree structure. This may mean that CUs within an I slice / tile group may consist of coding blocks of the Luma component or coding blocks of two Chroma components, and CUs within a P or B slice / tile group may consist of blocks of three color components. In this document, a slice may be referred to as a tile / tile group, and a tile / tile group may be referred to as a slice.

[0163] In the aforementioned "Partitioning of the CTUs using a tree structure," a quadtree coding tree structure involving a multitype tree was described, but the structure in which the CU is partitioned is not limited to this. For example, the BT structure and the TT structure can be interpreted as concepts included in the Multiple Partitioning Tree (MPT) structure, and the CU can be interpreted as being partitioned through the QT structure and the MPT structure. In an example where the CU is partitioned through the QT structure and the MPT structure, the partitioning structure can be determined by signaling a syntax element (e.g., MPT_split_type) containing information regarding how many blocks the leaf node of the QT structure is partitioned into, and a syntax element (e.g., MPT_split_mode) containing information regarding whether the leaf node of the QT structure is partitioned vertically or horizontally.

[0164] In another example, the CU may be divided in a way different from the QT structure, BT structure, or TT structure. That is, unlike when a lower-depth CU is divided into 1 / 4 the size of an upper-depth CU according to the QT structure, or a lower-depth CU is divided into 1 / 2 the size of an upper-depth CU according to the BT structure, or a lower-depth CU is divided into 1 / 4 or 1 / 2 the size of an upper-depth CU according to the TT structure, the lower-depth CU may, in some cases, be divided into 1 / 5, 1 / 3, 3 / 8, 3 / 5, 2 / 3, or 5 / 8 the size of an upper-depth CU, and the method of dividing the CU is not limited thereto.

[0165] Transformation / Inverse Transformation:

[0166] As described above, the encoding device can derive residual blocks (residual samples) based on blocks (predicted samples) predicted through intra / inter / IBC prediction, etc., and can derive quantized transformation coefficients by applying transformation and quantization to the derived residual samples. Information regarding the quantized transformation coefficients (residual information) can be included in the residual coding syntax and output in the form of a bitstream after encoding. The decoding device can obtain information regarding the quantized transformation coefficients (residual information) from the bitstream and derive the quantized transformation coefficients by decoding. The decoding device can derive residual samples by undergoing inverse quantization / inverse transformation based on the quantized transformation coefficients. As described above, at least one of quantization / inverse quantization and / or transformation / inverse transformation may be omitted. When the transform / inverse transform is omitted, the transform coefficients may be called coefficients or residual coefficients, or they may still be called transform coefficients for the sake of consistency in representation. Whether the transform / inverse transform is omitted can be signaled based on transform_skip_flag.

[0167] Transformation / inverse transformation can be performed based on transformation kernel(s). For example, according to this document, a multiple transform selection (MTS) scheme may be applied. In this case, some of the sets of multiple transformation kernels may be selected and applied to the current block. Transformation kernels may be referred to by various terms, such as transformation matrix or transformation type. For example, a set of transformation kernels may represent a combination of vertical transformation kernels and horizontal transformation kernels.

[0168] For example, MTS index information (or tu_mts_idx syntax elements) may be generated / encoded in an encoding device and signaled to a decoding device to indicate one of the sets of transformation kernels. For example, the sets of transformation kernels based on the values ​​of the MTS index information may be derived as shown in Table 2 (Specification of trTypeHor and trTypeVer depending on tu_mts_idx[ x ][ y ]), Table 3 (Specification of trTypeHor and trTypeVer depending on cu_sbt_horizontal_flag and cu_sbt_pos_flag), and / or Table 4 (Specification of trTypeHor and trTypeVer depending on predModeIntra).

[0169] [Table 2]

[0170]

[0171] The set of transformation kernels may be determined, for example, based on cu_sbt_horizontal_flag and cu__sbt_pos_flag.

[0172] If cu_sbt_horizontal_flag is 1, it indicates that the current coding unit is divided horizontally into two transformation units. If cu_sbt_horizontal_flag[ x0 ][ y0 ] is 0, it indicates that the current coding unit is divided vertically into two transformation units. If cu_sbt_pos_flag is 1, it indicates that tu_cbf_luma, tu_cbf_cb, and tu_cbf_cr of the first transformation unit of the current coding unit are not in the bitstream. If cu_sbt_pos_flag is 0, it indicates that tu_cbf_luma, tu_cbf_cb, and tu_cbf_cr of the second transformation unit of the current coding unit are not in the bitstream.

[0173] [Table 3]

[0174]

[0175] The set of transformation kernels may be determined, for example, based on the intra prediction mode for the current block.

[0176] [Table 4]

[0177]

[0178] In the tables above, trTypeHor can represent a horizontal direction conversion kernel, and trTypeVer can represent a vertical direction conversion kernel. Here, a trTypeHor / trTypeVer value of 0 can represent DCT2, a trTypeHor / trTypeVer value of 1 can represent DST7, and a trTypeHor / trTypeVer value of 2 can represent DCT8. However, this is merely an example, and by convention, other values ​​may be mapped to different DCTs / DSTs.

[0179] The following Table 5 (Transform basis functions of DCT-II / VIII and DSTVII for N-point input) illustrates exemplary basis functions for the aforementioned DCT2, DCT8, and DST7.

[0180] [Table 5]

[0181]

[0182] FIG. 15 shows the transform and inverse transform according to the embodiments.

[0183] In this document, the MTS-based transformation is applied as a primary transform, and a secondary transform may be applied. The secondary transform may be applied only to the coefficients in the upper-left wxh region of the coefficient block to which the primary transform is applied, and may be called the Reduced Secondary Transform (RST). For example, w and / or h may be 4 or 8. In the transformation, the primary and secondary transforms may be applied sequentially to the secondary block, and in the inverse transformation, the inverse secondary transform and the inverse primary transform may be applied sequentially to the transform coefficients. The secondary transform (RST transform) may be called the low frequency coefficients transform (LFCT) or low frequency non-separable transform (LFNST). The inverse secondary transform may be called the inverse LFCT or inverse LFNST.

[0184] FIG. 16 shows a Low-Frequency Non-Separable Transform (LFNST) according to embodiments.

[0185] The LFNST (Low Frequency Inseparable Transform), also known as the Reduced Secondary Transform, is applied between the forward first-order transform and quantization (encoder side), and between the inverse quantization and the inverse first-order transform (decoder side), as shown in FIG. 16. In the LFNST, a 4x4 inseparable transform or an 8x8 inseparable transform is applied depending on the block size. For example, a 4x4 LFNST is applied to small blocks (i.e., minimum value (width, height) < 8), and an 8x8 LFNST is applied to large blocks (i.e., minimum value (width, height) > 4).

[0186] The application of the inseparable transformation used in LFNST is explained as follows, using the input as an example. To apply a 4x4 LFNST, the 4x4 input block X is represented as a vector as follows.

[0187]

[0188]

[0189] Inseparable transformations are as follows: It is calculated as. Here represents the transformation coefficient vector, and T is a 16x16 transformation matrix. 16x1 coefficients The vector is then reconstructed into a 4x4 block using the scan order (horizontal, vertical, or diagonal) of the corresponding block. Factors with smaller indices are placed at smaller scan indices in the 4x4 factor block.

[0190] Transform / inverse transformation can be performed in units of CU or TU. That is, transformation / inverse transformation can be applied to residual samples within a CU or residual samples within a TU. The CU size and the TU size may be the same, or multiple TUs may exist within the CU area. Meanwhile, the term CU size generally refers to the luminance component (sample) CB size. The term TU size generally refers to the luminance component (sample) TB size. The chroma component (sample) CB or TB size can be derived based on the luminance component (sample) CB or TB size according to the component ratio based on the color format (chroma format, e.g., 4:4:4, 4:2:2, 4:2:0, etc.). The TU size can be derived based on maxTbSize. For example, if the CU size is greater than maxTbSize, multiple TU(TB) of maxTbSize are derived from the CU, and conversion / inverse conversion can be performed in TU(TB) units. maxTbSize can be considered for determining whether to apply various intra-prediction types, such as ISP. Information regarding maxTbSize may be determined in advance, or it may be generated and encoded by an encoding device and signaled to a decoding device.

[0191] Quantization / Dequantization:

[0192] As described above, the quantization unit of the encoding device can derive quantized conversion coefficients by applying quantization to conversion coefficients, and the inverse quantization unit of the encoding device or the inverse quantization unit of the decoding device can derive conversion coefficients by applying inverse quantization to quantized conversion coefficients.

[0193] In general, in video / image coding, the quantization rate can be varied, and compression can be adjusted using the varied quantization rate. From an implementation perspective, considering complexity, quantization parameters (QP) can be used instead of directly using the quantization rate. For example, quantization parameters can be integer values ​​from 0 to 63, and each quantization parameter value can correspond to an actual quantization rate. The quantization parameter (QPY) for the luminance component (luma sample) and the quantization parameter (QPC) for the chroma component (chroma sample) can be set differently.

[0194] The quantization process takes a transform coefficient (C) as input and divides it by a quantization rate (Qstep) to obtain a quantized transform coefficient (C'). In this case, considering computational complexity, the quantization rate can be multiplied by a scale to form an integer, and a shift operation can be performed by an amount corresponding to the scale value. A quantization scale can be derived based on the product of the quantization rate and the scale value. In other words, the quantization scale can be derived according to QP. Alternatively, the quantization scale can be applied to the transform coefficient (C) to derive the quantized transform coefficient (C').

[0195] The inverse quantization process is the reverse of the quantization process; by multiplying the quantized transformation coefficients (C') by the quantization rate (Qstep), the reconstructed transformation coefficients (C'') can be obtained based on this. In this case, a level scale can be derived depending on the quantization parameters, and the reconstructed transformation coefficients (C'') can be derived by applying this level scale to the quantized transformation coefficients (C''). The reconstructed transformation coefficients (C'') may differ slightly from the original transformation coefficients (C) due to losses during the transformation and / or quantization processes. Therefore, the encoding device performs inverse quantization in the same manner as the decoding device.

[0196] Meanwhile, adaptive frequency-weighted quantization technology, which adjusts the quantization intensity according to frequency, may be applied. Adaptive frequency-weighted quantization is a method of applying different quantization intensities for each frequency. Adaptive frequency-weighted quantization can apply different quantization intensities for each frequency by utilizing a predefined quantization scaling matrix. That is, the aforementioned quantization / de-quantization process can be performed based further on the quantization scaling matrix. For example, different quantization scaling matrices may be used depending on whether the prediction mode applied to the current block to generate the size of the current block and / or the residual signal of the current block is inter-prediction or intra-prediction. The quantization scaling matrix may be referred to as a quantization matrix or a scaling matrix. The quantization scaling matrix may be predefined. Additionally, for frequency-adaptive scaling, frequency-specific quantization scale information regarding the quantization scaling matrix may be configured / encoded in the encoding device and signaled to the decoding device. Frequency-specific quantization scale information can be referred to as quantization scaling information. Frequency-specific quantization scale information may include scaling list data (scaling_list_data). A (modified) quantization scaling matrix can be derived based on the scaling list data. Additionally, frequency-specific quantization scale information may include present flag information indicating the existence of scaling list data. Alternatively, it may further include information indicating whether scaling list data is modified at a lower level (e.g., PPS or tile group header, etc.) when scaling list data is signaled at a higher level (e.g., SPS).

[0197] Entropy Coding:

[0198] As described above in the description of FIG. 2, part or all of the video / image information may be entropied by the entropy encoding unit (240), and part or all of the video / image information described above in the description of FIG. 3 may be entropied by the entropy decoding unit (310). In this case, the video / image information may be encoded / decoded in units of syntax elements. In this document, the term "information is encoded / decoded" may include encoding / decoding by the method described in this paragraph.

[0199] FIG. 17 shows CABAC (Context Adaptive Binary Arithmetic Coding) encoding according to embodiments.

[0200] Figure 17 shows a block diagram of a CABAC for encoding a single syntax element. The encoding process of the CABAC first converts the input signal into a binary value through binarization if the input signal is a syntax element rather than a binary value. If the input signal is already a binary value, it is bypassed without undergoing binarization. Here, each binary digit 0 or 1 constituting the binary value is called a bin. For example, if the binary string after binarization (bin string) is 110, each of 1, 1, and 0 is called a bin. The bin(s) for a single syntax element can represent the value of the corresponding syntax element.

[0201] Binary bins are input into a regular coding engine or a bypass coding engine. The regular coding engine assigns a context model reflecting probability values ​​to the corresponding bin and encodes the bin based on the assigned context model. The regular coding engine can update the probability model for each bin after performing coding for it. Bins coded in this way are called context-coded bins. The bypass coding engine omits the procedure of estimating probabilities for input bins and the procedure of updating the probability model applied to the bin after coding. Instead of assigning context, it improves coding speed by coding input bins using a uniform probability distribution (e.g., 50:50). Bins coded in this way are called bypass bins. The context model can be assigned and updated per context-coded (regularly coded) bin, and the context model can be indicated based on ctxidx or ctxInc. ctxidx can be derived based on ctxInc. Specifically, for example, the context index (ctxidx) pointing to the context model for each normally coded bean can be derived as the sum of the context index increment (ctxInc) and the context index offset (ctxIdxOffset). Here, ctxInc can be derived differently for each bean. ctxIdxOffset can be represented as the lowest value of ctxIdx. The lowest value of ctxIdx can be called the initial value (initValue) of ctxIdx. ctxIdxOffset is a value generally used to distinguish context models for other syntax elements, and the context model for a single syntax element can be distinguished / derived based on ctxinc.

[0202] In the entropy encoding procedure, it is determined whether to perform encoding through a regular coding engine or a bypass coding engine, and the coding path can be switched. Entropy decoding performs the same process as entropy encoding in reverse order.

[0203] FIG. 18 illustrates an entropy encoding method according to embodiments.

[0204] The entropy coding described in Fig. 17 can be performed, for example, as shown in Fig. 18.

[0205] Referring to FIG. 18, an encoding device (entropy encoding unit) performs an entropy coding procedure regarding image / video information. The image / video information may include partitioning-related information, prediction-related information (e.g., inter / intra prediction distinction information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering-related information, etc., or may include various syntax elements related thereto. Entropy coding may be performed on a syntax element basis. S600 to S610 may be performed by the entropy encoding unit (240) of the encoding device of FIG. 2 described above.

[0206] The encoding device performs binarization on the target syntax element (S600). Here, the binarization may be based on various binarization methods, such as the Truncated Rice binarization process and the Fixed-length binarization process, and the binarization method for the target syntax element may be predefined. The binarization procedure may be performed by the binarization unit (242) within the entropy encoding unit (240).

[0207] The encoding device performs entropy encoding on the target syntax element (S610). The encoding device may encode the empty string of the target syntax element based on a regular coding-based (context-based) or bypass coding-based method, such as CABAC (context-adaptive arithmetic coding) or CAVLC (context-adaptive variable length coding), and the output may be included in a bitstream. The entropy encoding procedure may be performed by an entropy encoding processing unit (243) within the entropy encoding unit (240). As previously mentioned, the bitstream may be transmitted to a decoding device via a (digital) storage medium or a network.

[0208] FIG. 19 illustrates an entropy decoding method according to embodiments.

[0209] As shown in FIG. 19, a decoding device (entropy decoding unit) can decode encoded image / video information. The image / video information may include partitioning-related information, prediction-related information (e.g., inter / intra prediction distinction information, intra prediction mode information, inter prediction mode information, etc.), residual information, in-loop filtering-related information, etc., or may include various syntax elements related thereto. Entropy coding can be performed on a syntax element basis. S700 to S710 can be performed by the entropy decoding unit (310) of the decoding device of FIG. 3 described above.

[0210] The decoding device performs binarization on the target syntax element (S700). Here, the binarization may be based on various binarization methods, such as the Truncated Rice binarization process and the Fixed-length binarization process, and the binarization method for the target syntax element may be predefined. The decoding device may derive available empty strings (empty string candidates) for the available values ​​of the target syntax element through the binarization procedure. The binarization procedure may be performed by the binarization unit (312) within the entropy decoding unit (310).

[0211] The decoding device performs entropy decoding for the target syntax element (S710). The decoding device sequentially decodes and parses each bin for the target syntax element from the input bit(s) in the bitstream, and compares the derived bin string with the available bin strings for the corresponding syntax element. If the derived bin string is equal to one of the available bin strings, the value corresponding to the bin string is derived as the value of the corresponding syntax element. If not, the next bit in the bitstream is parsed further, and the procedure described above is performed again. Through this process, information (specific syntax element) can be signaled using variable-length bits without using start bits or end bits for specific information within the bitstream. Through this, relatively fewer bits can be allocated to low values, and overall coding efficiency can be increased.

[0212] The decoding device can decode each bin within a bin string from a bitstream in a context-based or bypass-based manner based on an entropy coding technique such as CABAC or CAVLC. The entropy decoding procedure can be performed by an entropy decoding processing unit (313) within the entropy decoding unit (310). As described above, the bitstream may contain various information for image / video decoding. As previously stated, the bitstream may be transmitted to the decoding device via a (digital) storage medium or a network.

[0213] In this document, a table containing syntax elements (syntax table) may be used to represent the signaling of information from an encoding device to a decoding device. The order of the syntax elements in the table containing syntax elements used in this document may represent the parsing order of the syntax elements from the bitstream. The encoding device may configure and encode the syntax table so that the syntax elements can be parsed by the decoding device in the parsing order, and the decoding device may obtain the values ​​of the syntax elements by parsing and decoding the syntax elements of the corresponding syntax table from the bitstream according to the parsing order.

[0214] FIG. 20 illustrates a picture decoding method according to embodiments.

[0215] General Video / Video Coding Procedures:

[0216] In video coding, the pictures constituting the video can be encoded / decoded according to a series of decoding orders. The picture order corresponding to the output order of the decoded pictures can be set differently from the decoding order, and based on this, not only forward prediction but also reverse prediction can be performed during inter-prediction.

[0217] FIG. 20 illustrates an example of a schematic picture decoding procedure to which the embodiment(s) of the present document are applicable. In FIG. 20, S900 may be performed in the entropy decoding unit (310) of the decoding device described in FIG. 3, S910 may be performed in the prediction unit (330), S920 may be performed in the residual processing unit (320), S930 may be performed in the addition unit (340), and S940 may be performed in the filtering unit (350). S900 may include the information decoding procedure described in the present document, S910 may include the inter / intra prediction procedure described in the present document, S920 may include the residual processing procedure described in the present document, S930 may include the block / picture restoration procedure described in the present document, and S940 may include the in-loop filtering procedure described in the present document.

[0218] As shown in FIG. 20, the picture decoding procedure may include, schematically as described in FIG. 3, a procedure for obtaining image / video information (through decoding) from a bitstream (S900), a picture restoration procedure (S910–S930), and an in-loop filtering procedure for the restored picture (S940). The picture restoration procedure may be performed based on prediction samples and residual samples obtained through the inter / intra prediction (S910) and residual processing (S920, inverse quantization and inverse transformation of quantized transformation coefficients) described in this document. A modified restored picture may be generated through an in-loop filtering procedure for the restored picture generated through the picture restoration procedure, and the modified restored picture may be output as a decoded picture and may also be stored in the decoded picture buffer or memory (360) of the decoding device and used as a reference picture in the inter prediction procedure when decoding the picture thereafter. In some cases, the in-loop filtering procedure may be omitted, in which case the restored picture may be output as a decoded picture and may also be stored in the decoded picture buffer or memory (360) of the decoding device and used as a reference picture in the inter-prediction procedure during subsequent decoding of the picture. The in-loop filtering procedure (S940) may include a deblocking filtering procedure, a sample adaptive offset (SAO) procedure, an adaptive loop filter (ALF) procedure, and / or a bilateral filter procedure, as described above, and some or all of these may be omitted. Additionally, one or some of the deblocking filtering procedure, the sample adaptive offset (SAO) procedure, the adaptive loop filter (ALF) procedure, and the bilateral filter procedure may be applied sequentially, or all of them may be applied sequentially. For example, the SAO procedure may be performed after the deblocking filtering procedure is applied to the restored picture.Alternatively, for example, the ALF procedure may be performed after a deblocking filtering procedure has been applied to the restored picture. This can be performed in the same way on the encoding device.

[0219] FIG. 21 illustrates a picture encoding method according to embodiments.

[0220] FIG. 21 illustrates an example of a schematic picture encoding procedure to which the embodiment(s) of the present document are applicable. In FIG. 21, S800 may be performed in the prediction unit (220) of the encoding device described above in FIG. 2, S810 may be performed in the residual processing unit (230), and S820 may be performed in the entropy encoding unit (240). S800 may include the inter / intra prediction procedure described in the present document, S810 may include the residual processing procedure described in the present document, and S820 may include the information encoding procedure described in the present document.

[0221] As shown in FIG. 21, the picture encoding procedure may include not only a procedure for encoding information for picture restoration (e.g., prediction information, residual information, partitioning information, etc.) in a general manner as described in FIG. 02 and outputting it in the form of a bitstream, but also a procedure for generating a restored picture for the current picture and a procedure for applying in-loop filtering to the restored picture (optional). The encoding device may derive (modified) residual samples from quantized transform coefficients through the inverse quantization unit (234) and the inverse transform unit (235), and may generate a restored picture based on the prediction samples and (modified) residual samples which are the outputs of S800. The restored picture thus generated may be identical to the restored picture generated by the decoding device described above. A modified restored picture can be generated through an in-loop filtering procedure for the restored picture, which can be stored in a decoded picture buffer or memory (270), and, as in the case of a decoding device, can be used as a reference picture in an inter-prediction procedure during the encoding of the picture thereafter. As described above, in some cases, part or all of the in-loop filtering procedure may be omitted. When the in-loop filtering procedure is performed, filtering-related information (parameters) can be encoded in the entropy encoding unit (240) and output in the form of a bitstream, and the decoding device can perform the in-loop filtering procedure in the same way as the encoding device based on the filtering-related information.

[0222] Through this in-loop filtering procedure, noise generated during video coding, such as blocking and ringing artifacts, can be reduced, and subjective and objective visual quality can be enhanced. Furthermore, by performing the in-loop filtering procedure in both the encoding and decoding devices, they can derive identical prediction results, increase the reliability of picture coding, and reduce the amount of data that must be transmitted for picture coding.

[0223] As described above, the picture restoration procedure can be performed not only in the decoding device but also in the encoding device. Restoration blocks can be generated based on intra-prediction / inter-prediction on a block-by-block basis, and a restored picture containing the restoration blocks can be generated. If the current picture / slice / tile group is the I picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based solely on intra-prediction. Meanwhile, if the current picture / slice / tile group is the P or B picture / slice / tile group, the blocks included in the current picture / slice / tile group can be restored based on intra-prediction or inter-prediction. In this case, inter-prediction may be applied to some blocks within the current picture / slice / tile group, and intra-prediction may be applied to some remaining blocks. The color components of the picture may include luminance components and chroma components, and unless explicitly limited in this document, the methods and embodiments proposed in this document may be applied to luminance components and chroma components.

[0224] Examples of coding hierarchy and structure:

[0225] The coded video / image according to this document can be processed according to, for example, the coding layers and structures described below.

[0226] FIG. 22 shows a hierarchical structure for a coded image according to embodiments.

[0227] FIG. 22 is a diagram illustrating the hierarchical structure of a coded image.

[0228] The coded video is divided into a video coding layer (VCL) that handles the decoding processing of the video and the video itself, a subsystem that transmits and stores the encoded information, and a network abstraction layer (NAL) that exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0229] In VCL, VCL data containing compressed image data (slice data) can be generated, or parameter sets containing information such as Picture Parameter Set (PPS), Sequence Parameter Set (SPS), and Video Parameter Set (VPS), or SEI (Supplemental Enhancement Information) messages that are additionally required in the decoding process of the image can be generated.

[0230] In NAL, a NAL unit can be created by adding header information (NAL unit header) to the Raw Byte Sequence Payload (RBSP) generated in VCL. In this case, the RBSP refers to slice data, parameter sets, SEI messages, etc. generated in VCL. The NAL unit header may include NAL unit type information specified according to the RBSP data included in the NAL unit.

[0231] NAL units can be classified into VCL NAL units and Non-VCL NAL units depending on the RBSP generated in VCL. A VCL NAL unit may refer to a NAL unit containing information about an image (slice data), and a Non-VCL NAL unit may refer to a NAL unit containing information necessary to decode an image (parameter set or SEI message).

[0232] The aforementioned VCL NAL unit and Non-VCL NAL unit can be transmitted over a network by attaching header information according to the data specifications of the underlying system. For example, the NAL unit can be transformed into a data format of a specified specification, such as H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

[0233] The NAL unit type can be determined according to the RBSP data structure included in the NAL unit, and information about this NAL unit type can be stored in the NAL unit header and signaled.

[0234] For example, NAL units can be broadly classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether they contain information about the image (slice data). VCL NAL unit types can be classified according to the properties and types of the picture included in the VCL NAL unit, while Non-VCL NAL unit types can be classified according to the types of parameter sets.

[0235] The following is an example of a NAL unit type specified based on the type of parameter set included by the Non-VCL NAL unit type: APS (Adaptation Parameter Set) NAL unit: A type for a NAL unit containing APS. DPS (Decoding Parameter Set) NAL unit: A type for a NAL unit containing DPS. VPS (Video Parameter Set) NAL unit: A type for a NAL unit containing VPS. SPS (Sequence Parameter Set) NAL unit: A type for a NAL unit containing SPS. PPS (Picture Parameter Set) NAL unit: A type for a NAL unit containing PPS.

[0236] The above-described NAL unit types have syntax information for the NAL unit type, and the syntax information can be stored in the NAL unit header and signaled. For example, the syntax information can be nal_unit_type, and the NAL unit types can be specified by the nal_unit_type value.

[0237] A slice header (slice header syntax) may include information / parameters that can be applied commonly to slices. An APS (APS syntax) or a PPS (PPS syntax) may include information / parameters that can be applied commonly to one or more slices or pictures. An SPS (SPS syntax) may include information / parameters that can be applied commonly to one or more sequences. A VPS (VPS syntax) may include information / parameters that can be applied commonly to multiple layers. A DPS (DPS syntax) may include information / parameters that can be applied commonly across the video. A DPS may include information / parameters related to the concatenation of a CVS (coded video sequence). In this document, High-level syntax (HLS) may include at least one of an APS syntax, a PPS syntax, an SPS syntax, a VPS syntax, a DPS syntax, and a slice header syntax.

[0238] In this document, the image / video information that is encoded from an encoding device to a decoding device and signaled in the form of a bitstream includes not only information related to partitioning within a picture, intra / inter prediction information, residual information, in-loop filtering information, etc., but may also include information included in a slice header, information included in an APS, information included in the PPS, information included in an SPS, and / or information included in a VPS.

[0239] Coding descriptors:

[0240] The following descriptors represent the parsing process for each syntax element: ae(v): Context-adaptive arithmetic entropy-coded syntax element. b(8): A byte containing a bit string of arbitrary patterns (8 bits). The parsing process for this descriptor is specified by the return value of the read_bits(8) function. f(n): A fixed-pattern bit string of n bits written from left to right with the left bit coming first. The parsing process for this descriptor is specified by the return value of the read_bits(n) function. i(n): A signed integer using n bits. If n is "v" in the syntax table, the number of bits depends on the values ​​of other syntax elements. The parsing process for this descriptor is specified by the return value of the read_bits(n) function and is interpreted as a two's complement integer representation with the most significant bit written first. se(v): A signed integer zero-ordered Exp-Golomb-coded syntax element, with the left bit coming first. The parsing process for this descriptor is specified by the order of k being zero. st(v): A null-terminated string encoded in Universal Coded Character Set (UCS) Transfer Format-8 (UTF-8) characters as specified in ISO / IEC 10646. The parsing process is as follows: st(v) moves the bitstream pointer (stringLength + 1) * 8 bit positions starting from the current position in the bitstream's byte alignment position to the next byte alignment byte, such as 0x00 (excluding that byte), where stringLength is equal to the number of bytes returned. The st(v) syntax descriptor is used in this specification only when the current position in the bitstream is the byte alignment position. tu(v): A truncated unary operator using up to maxVal bits. maxVal is defined in the semantics of the symtax ​​element. u(n): An unsigned integer using n bits.In the syntax table, if n is "v", the number of bits depends on the values ​​of other syntax elements. The parsing process of this descriptor is specified by the return value of the function read_bits(n) and is interpreted as a binary representation of an unsigned integer with the most significant bit written first. ue(v): An unsigned integer of a zero-order exponent Colomb-coded syntax element with the left bit written first. The parsing process of this descriptor is specified by setting the order of k to 0.

[0241] High-level syntax signaling and semantics are described below with reference to each figure.

[0242] FIGS. 23a, FIGS. 23b, FIGS. 23c, FIGS. 23d, and FIGS. 23e show picture header structures according to embodiments.

[0243] Picture header and slice header:

[0244] A coded picture may consist of one or more slices. Parameters describing the coded picture are passed within the picture header (PH), and parameters describing the slice are passed within the slice header. The PH is passed as its own NAL unit type. The SH is located at the beginning of the NAL unit containing the slice's payload (e.g., slice data). For details on the syntax and semantics of the PH and SH, refer to Section 7 of the VVC specification.

[0245] SEI Messages:

[0246] Neural-network post-filter SEI messages

[0247] General post-processing filtering processes using NNPFs

[0248]

[0249] The input to this process is a bitstream BitstreamToFilter. The output of this process is a list of NNPF output pictures, ListNnpfOutputPics.

[0250] First, BitstreamToFilter is decoded, and the CroppedDecodedPictures list is set as a list of decoded pictures cropped in the order of the BitstreamToFilter decoding output.

[0251] Second, a filtering process for one picture is in CroppedDecodedPictures and is repeatedly called in output order for each cropped decoded picture with one or more NNPFs enabled.

[0252] The order of the pictures in ListNnpfOutputPics is the output order.

[0253] There must be only one picture associated with a specific output time instance within ListNnpfOutputPics. If there are multiple NNPFs enabled for a specific picture in CroppedDecodedPictures and only one NNPF can be selected to apply (other NNPFs can also be selected), the above constraint applies regardless of which NNPF is applied to the specific picture.

[0254] Single Picture Filtering Process Using NNPF:

[0255] The filtering process is applied to each cropped decoded picture (referred to as the current picture) that belongs to CroppedDecodedPictures and has one or more NNPFs enabled.

[0256] When applying NNPF to the current picture, the filtered and / or interpolated picture is generated by NNPF by applying the NNPF process specified in the semantics of the NNPFFC SEI message to the current picture in a patch manner.

[0257] When applying NNPF to the current picture, the order of the picture generated by NNPF by applying the NNPF process is the same as the output order stored in the output tensor of NNPF.

[0258] If the applied NNPF is the last NNPF applied to the current picture, the picture generated by the NNPF and the picture output from the NNPF process are included in ListNnpfOutputPics, in the same order as when the picture is stored in the output tensor of the NNPF.

[0259] The image encoding method, image encoding device, image decoding method, image decoding device, method for transmitting a bitstream, and recording medium storing a bitstream according to embodiments include a method and device for a digitally signed content SEI message.

[0260] This disclosure describes Digitally Signed Content SEI messages. The described method is based on Versatile Video Coding (VVC) and Versatile Supplemental Enhancement Information Messages (VSEI) for coded video bitstreams, but may be applicable to other video coding technologies.

[0261] The present disclosure relates to the following.

[0262] (1) Versatile Video Coding (VVC). The latest VVC specification can be found at: [https: / jvet-experts.org / doc_end_user / documents / 20_Teleconference / wg11 / JVET-T2001-v2.zip](https: / jvet-experts.org / doc_end_user / documents / 20_Teleconference / wg11 / JVET-T2001-v2.zip)

[0263] (2) Versatile supplemental enhancement information messages (VSEI) for coded video bitstreams. The latest VSEI specification can be found at: [https: / www.itu.int / rec / T-REC-H.274](https: / www.itu.int / rec / T-REC-H.274)

[0264] (3) Additional SEI messages for VSEI (Draft 3). The latest draft is available at: [https: / jvet-experts.org / doc_end_user / documents / 31_Geneva / wg11 / JVET-AE2006-v2.zip](https: / jvet-experts.org / doc_end_user / documents / 31_Geneva / wg11 / JVET-AE2006-v2.zip)

[0265] (4) Technology to Consider for Future Expansion of VSEI (Draft 6): [https: / jvet-experts.org / doc_end_user / documents / 36_Kemer / wg11 / JVET-AJ2032-v2.zip](https: / jvet-experts.org / doc_end_user / documents / 36_Kemer / wg11 / JVET-AJ2032-v2.zip)

[0266] (5) SEI messages for VSEI version 4 (Draft 4): [https: / jvet-experts.org / doc_end_user / documents / 36_Kemer / wg11 / JVET-AJ2006-v2.zip](https: / jvet-experts.org / doc_end_user / documents / 36_Kemer / wg11 / JVET-AJ2006-v2.zip)

[0267] An encoding device performing an encoding method according to embodiments performs the source device of FIG. 1, an encoder of FIG. 2, a system of FIG. 4, encoding according to the method of FIG. 5 to FIG. 22, generation of SEI messages such as FIG. 23 to FIG. 26, a reference method for verifying the integrity of a Temporal ID-related substream of FIG. 27, and a method of FIG. 28.

[0268] A decoding device performing a decoding method according to embodiments performs the receiving device of FIG. 1, a decoder of FIG. 3, a system of FIG. 4, decoding according to the method of FIG. 5 to FIG. 22, acquisition of SEI messages such as FIG. 23 to FIG. 26, a reference method for verifying the integrity of a Temporal ID-related substream of FIG. 27, and the method of FIG. 29.

[0269] SEI messages according to the embodiments may include the following SEI messages. For example, they may include a digitally signed content initialization SEI message.

[0270] FIG. 24 shows the digitally signed content initialization SEI message syntax according to the embodiments.

[0271] The semantics of the digitally signed content initialization SEI message of Fig. 24 are as follows:

[0272] To use the digitally signed content initialization SEI message, the following must be defined:

[0273] nonVclDigitallySignedNalUnitsList, a list of non-VCL NAL unit type identifiers.

[0274] The digitally signed content initialization SEI message, the digitally signed content selection SEI message, and the digitally signed content verification SEI message provide a mechanism to verify that a video coded by a content provider identifying itself through the digital certificate referenced in the digitally signed content initialization SEI message has been generated. The digitally signed content initialization SEI message also provides information regarding a secure hash algorithm used to calculate a message digest used in conjunction with the digital signatures present in the digitally signed content verification SEI messages, in order to verify the trustworthiness of non-VCL NAL units having a NAL unit type identifier that is one of the values ​​in nonVclDigitallySignedNalUnitsList, and VCL NAL units present in the coded video sequence. Additionally, it provides information regarding the digital signature algorithm used and the content provider's public key. A digitally signed content initialization SEI message may provide the content provider's public key by providing a URI (as defined in ISO / IEC 21617-1) that identifies a trust record containing the content provider's certificate, or by providing a URI that directly identifies the certificate.

[0275] If a digitally signed content initialization SEI message exists for any AU of the CVS, then a digitally signed content initialization SEI message must exist for all IDRs, CRAs, and GDR PUs of the CVS.

[0276] If a digitally signed content initialization SEI message exists in any AU of CVS, it must precede all non-VCL NAL units and VCL NAL units of said AU having a NAL unit type identifier that is one of the values ​​in nonVclDigitallySignedNalUnitsList.

[0277] The digitally signed content initialization SEI message applies to the currently coded picture and all subsequent coded pictures until one or more of the following conditions are true:

[0278] - The bitstream ends.

[0279] - A new CVS begins.

[0280] - A new digitally signed content initialization SEI message is received.

[0281] If a digitally signed content initialization SEI message exists in the CVS AU, a digitally signed content verification SEI message must exist for each substream to which an NAL unit is allocated. The signed content verification SEI message must exist in the bitstream before one or more of the following conditions become true:

[0282] - The bitstream ends.

[0283] - A new CVS begins.

[0284] - A new digitally signed content initialization SEI message is received.

[0285] The hash method type (dsci_hash_method_type) represents a secure hash algorithm used to compute a message digest for a subset of non-VCL NAL units having a NAL unit type identifier that is one of the values ​​in nonVclDigitallySignedNalUnitsList, and for VCL NAL units of the encoded video sequence. Based on this message digest and the digital signature present in the digitally signed content verification SEI messages, the decoder can verify that the video encoded was created by the content originator indicated by dsci_key_register_idx when the dsci_use_key_register_idx_flag flag is 1. Supported values ​​for the syntactic element dsci_hash_method_type, the block size used for computing the message digest, and the size of the computed message digest are specified. dsci_hash_method_type values ​​not listed in Table 6 (Supported values ​​of dsci_hash_method_type) are reserved for future use by ITU-T | ISO / IEC and must not exist in payload data suitable for this version of the specification. The decoder must ignore trustworthy initialization SEI messages containing values ​​reserved for dsci_hash_method_type. The secure hash algorithms listed in Table 6 are specified in the "Secure Hash Standard" NIST FIPS PUB 180-4.

[0286] [Table 6]

[0287]

[0288] The key source URL (dsci_key_source_uri) contains a URI having the syntax and semantics defined in IETF Internet Standard 66. If dsci_key_retrieval_mode_idc is equal to 0, dsci_key_source_uri specifies a trust record defined in ISO / IEC 21617-1. If dsci_key_retrieval_mode_idc is equal to 1, the following applies:

[0289] If the use key register index flag (dsci_use_key_register_idx_flag) is equal to 0, the URI identifies a content provider certificate that can be used to verify signatures present in subsequent digitally signed content verification SEI messages;

[0290] In other cases (i.e., when dsci_use_key_register_idx_flag is equal to 1), the URI identifies a register of certificates containing a content provider's certificate as directed by dsci_key_register_idx, which can be used to verify signatures present in subsequent digitally signed content verification SEI messages.

[0291] The value obtained by adding 1 to the number of verification substreams (dsci_num_verification_substreams_minus1) indicates the number of substreams in which the signature may exist in subsequent digitally signed content verification SEI messages after the message digest is calculated.

[0292] The variable NumVerificationSubstream is derived as follows:

[0293] NumVerificationSubstream = dsci_num_verification_substreams_minus1 + 1.

[0294] A key retrieval mode indicator (dsci_key_retrieval_mode_idc) equal to 0 indicates that the URI contained in dsci_key_source_uri specifies a trust record as defined in ISO / IEC 21617-1. A dsci_key_retrieval_mode_idc equal to 1 indicates that the URI contained in dsci_key_source_uri and, if present, dsci_key_register_idx specify a certificate. In this version of the specification, dsci_key_retrieval_mode_idc must be in the range of 0 to 1. The decoder must also accept other values ​​of dsci_key_retrieval_mode_idc, provided that it ignores the contents of digitally signed content initialization SEI messages, associated digitally signed content selection SEI messages, and associated digitally signed content verification SEI messages.

[0295] A use key register index flag (dsci_use_key_register_idx_flag) equal to 1 indicates that the URI contained in dsci_key_source_uri specifies a register of certificates and that the syntax element dsci_key_register_idx exists within the SEI message. A dsci_use_key_register_idx_flag equal to 0 indicates that the URI contained in dsci_key_source_uri specifies a certificate and that the syntax element dsci_key_register_idx does not exist within the SEI message.

[0296] If the key retrieval mode indicator (dsci_key_retrieval_mode_idc) is equal to 0, the media asset for which the last trust manifest within the trust record specified in ISO / IEC 21617-1 provides content binding is a digitally signed content initialization SEI message. The following constraints apply to the trust record specified in ISO / IEC 21617-1 identified by dsci_key_source_uri:

[0297] - The last trust manifest within the trust record specified in ISO / IEC 21617-1 must contain exactly one hard binding data hash assertion with the same label as c2pa.hash.data.

[0298] - The schema for data hash assertions is defined by the data-hash-map rule in the following CDDL definition:

[0299] cddl

[0300] ; It is a data structure used to store cryptographic hashes for part or all of the asset data, and

[0301] ; It also includes additional information needed to calculate the hash.

[0302] data-hash-map = {

[0303] ? "exclusions": [1* EXCLUSION_RANGE-map], ; ranges must have a `start` value that is monotonically increasing, and no two ranges can overlap.

[0304] ? "alg": tstr .size (1..max-tstr-length), ; A string identifying the cryptographic hash algorithm used to calculate the hash in this assertion

[0305] "hash": bstr, ; byte string of the hash value

[0306] "pad": bstr, ; A zero-padded byte string used to fill spaces

[0307] ? "pad2": bstr, ; Optional zero-padded byte string used to fill spaces

[0308] ? "name": tstr .size (1..max-tstr-length), ; (Optional) A human-readable string describing the range covered by this hash

[0309] ? "url": uri, ; deprecated and discarded.

[0310] }

[0311] EXCLUSION_RANGE-map = {

[0312] "start": int, ; Starting byte of the range

[0313] "length": int, ; Number of bytes of data to exclude

[0314] }

[0315] - The exclusion range, which specifies the data (exclusion range) within the digitally signed content initialization SEI message that is indicated in the data hash assertion and excluded when calculating the hash value, must match the dsci_key_source_uri bytes within the digitally signed content initialization SEI message.

[0316] dsci_key_register_idx (if present) contains an index specifying the content provider's certificate within the certificate register indicated by dsci_key_source_uri, which can be used to verify the signature included in subsequent digitally signed content verification SEI messages.

[0317] The key retrieval mode indicator (dsci_key_retrieval_mode_idc), the use key register index flag (dsci_use_key_register_idx_flag), the key source URL (dsci_key_source_uri), and the certificate indicated by the key register index (dsci_key_register_idx) when the use key register index flag (dsci_use_key_register_idx_flag) is 1 must specify the digital signature method and the content provider's public key accompanied by relevant parameters (where applicable). When dsci_key_retrieval_mode_idc is 1, the format in which this information is provided is outside the scope of this specification. It is recommended to use a digital signature algorithm compliant with NIST FIPS 186-5, the "Digital Signature Standard."

[0318] If the content uuid present flag (dsci_content_uuid_present_flag) is 1, it means that the syntax element dsci_content_uuid exists, and if dsci_content_uuid_present_flag is 0, it means that dsci_content_uuid does not exist. If dsci_key_retrieval_mode_idc is 0, dsci_content_uuid_present_flag must be 1.

[0319] The content uuid (dsci_content_uuid) represents an identifier for the video content (if present) and must have a value specified as a UUID according to the procedure of ISO / IEC 11578:1996.

[0320] If a digitally signed content initialization SEI message exists within the AU, NumVerificationSubstream message digests are initialized for the specified dsci_hash_method_type according to the specifications defined in NIST FIPS PUB 180-4. Each non-VCL NAL unit having a NAL unit type identifier corresponding to one of the values ​​in nonVclDigitallySignedNalUnitsList, and VCL NAL units following the digitally signed content initialization SEI message, are associated with one of the NumVerificationSubstream message digests. The verification substream ID is indicated by the digitally signed content selection SEI message, or is presumed to be 0 if no digitally signed content selection SEI message exists for a specific PU.

[0321] When k is in the range from 0 to dsci_num_verification_substreams_minus1 (inclusive), the message used to calculate the k-th message digest is obtained by concatenating all non-VCL NAL units with NAL unit type identifiers corresponding to one of the values ​​in nonVclDigitallySignedNalUnitsList and all VCL NAL units associated with the k-th verification substream. The calculation of the message digest is performed in blocks, and the block size is specified by the dsci_hash_method_type value. For each non-VCL NAL unit and each VCL NAL unit with a NAL unit type identifier corresponding to one of the values ​​in nonVclDigitallySignedNalUnitsList, the associated message digest is updated according to the algorithm specified in NIST FIPS PUB 180-4 for the specified dsci_hash_method_type.

[0322] It should be noted that since the message digest is calculated for data concatenating all non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to one of the values ​​included in nonVclDigitallySignedNalUnitsList for a specific validation substream, some processing blocks may typically span two or more consecutive NAL units.

[0323] FIG. 25 shows a digitally signed content selection SEI message syntax according to embodiments.

[0324] The SEI message according to the embodiments includes a digitally signed content selection SEI message.

[0325] A digitally signed content selection SEI message may include a verification substream ID (dscs_verification_substream_id).

[0326] The semantics of the SEI message for selecting digitally signed content are as follows.

[0327] To use this SEI message, the following definition is required.

[0328] - nonVclDigitallySignedNalUnitsList, a list of non-VCL NAL unit type identifiers.

[0329] The digitally signed content selection SEI message provides a mechanism for associating a coded picture with one of the verification substreams indicated in the digitally signed content initialization SEI message.

[0330] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content selection SEI message, the digitally signed content initialization SEI message must precede the digitally signed content selection SEI message in the decoding order.

[0331] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content selection SEI message.

[0332] If a digitally signed content selection SEI message exists in any AU of CVS, it must precede all non-VCL NAL units and VCL NAL units of the AU that have a NAL unit type identifier corresponding to one of the values ​​included in the nonVclDigitallySignedNalUnitsList of that AU.

[0333] The verification substream ID (dscs_verification_substream_id) represents the verification substream to which non-VCL NAL units and VCL NAL units are assigned, having NAL unit type identifiers corresponding to one of the values ​​included in the nonVclDigitallySignedNalUnitsList of the current coded picture. If a digitally signed content initialization SEI message exists for the current coded video sequence but a digitally signed content selection SEI message does not exist for a coded picture, the value of dscs_verification_substream_id is presumed to be 0. The value of dscs_verification_substream_id must be in the range from 0 to dsci_num_verification_substreams_minus1 (inclusive).

[0334] The message digest for the verification substream with id dscs_verification_substream_id is updated with non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to one of the values ​​included in the nonVclDigitallySignedNalUnitsList of the currently encoded image, according to the dsci_hash_method_type specified in the preceding digitally signed content initialization SEI message.

[0335] FIG. 26 shows the digitally signed content verification SEI message syntax according to the embodiments.

[0336] The SEI message according to the embodiments includes digitally signed content verification SEI message syntax.

[0337] The use of digitally signed content verification SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0338] The digitally signed content verification SEI message provides a mechanism for verifying the digital signature of video content.

[0339] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content verification SEI message.

[0340] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content verification SEI message, the digitally signed content initialization SEI message must be placed in decoding order prior to the digitally signed content verification SEI message. Additionally, if the PU includes both a digitally signed content selection SEI message and a digitally signed content verification SEI message, the digitally signed content selection SEI message must be placed in decoding order prior to the digitally signed content verification SEI message.

[0341] If a digitally signed content verification SEI message exists in the PU of the CVS, a non-VCL NAL unit or a VCL NAL unit having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList must not be assigned to the substream indicated by dscv_verification_substream_id unless one or more of the conditions are met, such as the bitstream being terminated, a new CVS being started, or a new digitally signed content initialization SEI message being received.

[0342] The verification substream ID (dscv_verification_substream_id) indicates the verification substream to which the above SEI message applies.

[0343] Signature length (dscv_signature_length_in_octets_minus1): This value plus 1 (i.e., dscv_signature_length_in_octets_minus1+1) specifies the length of the syntax element dscv_signature in octets, where 1 octet consists of 8 bits.

[0344] The signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id. Verification of the bitstream signature consists of the following sequential steps.

[0345] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: the concatenation of non-VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList and VCL NAL units for a verification substream with id equal to dscv_verification_substream_id is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 7 (composition of the identification string (IdString)).

[0346] (2) The reference message digest RefDigest is determined as follows.

[0347] If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id.

[0348] In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI message (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 7.

[0349] Otherwise, the reference message digest RefDigest is the last calculated message digest for the validation substream with id 0.

[0350] (3) The identification string IdString is constructed by concatenating the binary representations of the reference message digest RefDigest, the current message digest, dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 7.

[0351] [Table 7]

[0352]

[0353] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0354] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0355] NOTE 1 ― Since the bitstring used for signature verification includes RefDigest, it is possible to verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList used in the calculation of the current message digest are correct, and further verify that non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the additional values ​​of nonVclDigitallySignedNalUnitsList were not added to the bitstream, and that non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList were not removed from the bitstream.

[0356] NOTE 2 ― When the decoder is tuned to the bitstream, the value of RefDigest cannot be calculated accurately, so the IdString configured for the first digitally signed content verification SEI message cannot be verified. However, starting from the second digitally signed content verification SEI message, the signatures can be verified.

[0357] After verification, message digests for verification substreams with the same ID as dscv_verification_substream_id are reinitialized for the specified dsci_hash_method_type in accordance with the NIST FIPS PUB 180-4 specification.

[0358] FIG. 27 shows the current structure of RefDigest according to the embodiments.

[0359] FIG. 27 shows a RefDigest structure related to the SEI messages described in FIG. 24 to 26, etc.

[0360] Regarding the technical problem to be solved in the embodiments, digitally signed content SEI messages (DSC) are currently included in VSEI version 4 (JVET-AJ2006). Digitally signed content SEI messages consist of three types and include an Initialization SEI that establishes a verification context for CVS, a Selection SEI that assigns NAL units to verification substreams, and a Verification SEI that verifies substream integrity using current and reference digests. RefDigest ensures temporal consistency by referencing the hash value (Message Digest) of the substream identified by dscv_verification_substream_id - 1, and always points to the immediately preceding substream. This approach works effectively for simple and continuous substream structures, but poses challenges in scenarios involving non-contiguous substreams or hierarchical scalability.

[0361] Figure 27 shows the current structure of RefDigest, and arrows indicate RefDigest referenced by each substream.

[0362] The current structure allows substream substrA to reference another substream substrB without restrictions on hierarchical relationships, and consequently, the following problems arise.

[0363] (1) Hierarchical Inconsistencies: As illustrated in FIG. 27, substream 3 refers to substream 2, which contains pictures of a higher layer ID or temporal ID. This conflicts with the expected hierarchical order, which dictates that lower substreams should not refer to higher layers or temporal levels.

[0364] (2) Scalability Disruption: Allowing substream 3 of a lower layer or temporal sublayer to reference substream 2 of a higher layer or sublayer disrupts the hierarchy required for scalability. Such references hinder the efficient adaptation of substreams for specific use cases, such as partial extraction or random access.

[0365] Example 1

[0366] The use of digitally signed content initialization SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0367] The digitally signed content initialization SEI message, the digitally signed content selection SEI message, and the digitally signed content verification SEI message provide a mechanism to verify that the coded video was generated by a content provider that identifies itself through a digital certificate referenced in the digitally signed content initialization SEI message. Additionally, these SEI messages provide information regarding a secure hash algorithm used to calculate a message digest, and said message digests are used in conjunction with the digital signatures present in the digitally signed content verification SEI messages to verify the trustworthiness of non-VCL NAL units having a NAL unit type identifier corresponding to any one of the values ​​in nonVclDigitallySignedNalUnitsList, and the VCL NAL units present in the coded video sequence. In addition, this SEI message provides further information regarding the digital signature algorithm used and the content provider's public key. The digitally signed content initialization SEI message may provide the content provider's public key by providing a URI that identifies a trust record containing the content provider's certificate (as specified in ISO / IEC 21617-1) or by providing a URI that directly identifies the certificate.

[0368] If a digitally signed content initialization SEI message exists in any AU of the CVS, then a digitally signed content initialization SEI message must exist for all IDRs, CRAs, and GDR PUs of the CVS.

[0369] If a digitally signed content initialization SEI message exists in any AU of CVS, said message must precede all non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList of said AU.

[0370] The digitally signed content initialization SEI message applies to the current coded picture and all subsequent coded pictures until one or more of the following conditions are met: the bitstream ends, a new CVS begins, or a new digitally signed content initialization SEI message is received.

[0371] If a digitally signed content initialization SEI message exists in the AU of the CVS, a digitally signed content verification SEI message must exist for each substream to which a NAL unit is allocated. The said digitally signed content verification SEI message must exist in the bitstream before one or more of the conditions are met, such as the bitstream ending, a new CVS starting, or a new digitally signed content initialization SEI message being received.

[0372] Verification substreams associated with the same initialization SEI message may consist of NAL units belonging to the same layer and / or temporal sublayer.

[0373] For example, substream 3 can reference substream 0, which belongs to the same layer. Substream 4 can reference substream 1, which belongs to the same layer. Substream 5 can reference substream 2, which belongs to the same layer.

[0374] Example 2

[0375] The use of digitally signed content initialization SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0376] The digitally signed content initialization SEI message, the digitally signed content selection SEI message, and the digitally signed content verification SEI message provide a mechanism to verify that the coded video was generated by a content provider that identifies itself through a digital certificate referenced in the digitally signed content initialization SEI message. Additionally, these SEI messages provide information regarding a secure hash algorithm used to calculate a message digest, and said message digests are used in conjunction with the digital signatures present in the digitally signed content verification SEI messages to verify the trustworthiness of non-VCL NAL units having a NAL unit type identifier corresponding to any one of the values ​​in nonVclDigitallySignedNalUnitsList, and the VCL NAL units present in the coded video sequence. In addition, this SEI message provides further information regarding the digital signature algorithm used and the content provider's public key. The digitally signed content initialization SEI message may provide the content provider's public key by providing a URI that identifies a trust record containing the content provider's certificate (as specified in ISO / IEC 21617-1) or by providing a URI that directly identifies the certificate.

[0377] If a digitally signed content initialization SEI message exists in any AU of the CVS, then a digitally signed content initialization SEI message must exist for all IDRs, CRAs, and GDR PUs of the CVS.

[0378] If a digitally signed content initialization SEI message exists in any AU of CVS, said message must precede all non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList of said AU.

[0379] The digitally signed content initialization SEI message applies to the current coded picture and all subsequent coded pictures until one or more of the following conditions are met: the bitstream ends, a new CVS begins, or a new digitally signed content initialization SEI message is received.

[0380] If a digitally signed content initialization SEI message exists in the AU of the CVS, a digitally signed content verification SEI message must exist for each substream to which a NAL unit is allocated. The said digitally signed content verification SEI message must exist in the bitstream before one or more of the conditions are met, such as the bitstream ending, a new CVS starting, or a new digitally signed content initialization SEI message being received.

[0381] Verification substreams associated with the same initialization SEI message must consist of NAL units belonging to the same layer and / or temporal sublayer.

[0382] For example, substream 3 can reference substream 0, which belongs to the same layer. Substream 4 can reference substream 1, which belongs to the same layer. Substream 5 can reference substream 2, which belongs to the same layer.

[0383] Example 3

[0384] The use of a digitally signed content selection SEI message requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0385] The digitally signed content selection SEI message provides a mechanism for associating coded pictures with one of the verification substreams indicated in the digitally signed content initialization SEI message.

[0386] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content selection SEI message, the digitally signed content initialization SEI message must be placed in decoding order prior to the digitally signed content selection SEI message.

[0387] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content selection SEI message.

[0388] If a digitally signed content selection SEI message exists in any AU of CVS, said message must precede all non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList of said AU.

[0389] The verification substream ID (dscs_verification_substream_id) indicates the verification substream to which non-VCL NAL units with NAL unit type identifiers corresponding to any of the values ​​in nonVclDigitallySignedNalUnitsList and VCL NAL units of the current coded picture are assigned. If a digitally signed content initialization SEI message exists for the current coded video sequence but a digitally signed content selection SEI message does not exist for the coded picture, the value of dscs_verification_substream_id is inferred to be equal to 0. The value of dscs_verification_substream_id must be within the range (inclusive) from 0 to dsci_num_verification_substreams_minus1. Verification substreams indicated by sequential dscs_verification_substream_id may consist of NAL units belonging to the same layer and / or temporal sublayer.

[0390] The message digest for the verification substream with id equal to dscs_verification_substream_id is updated using non-VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList and the VCL NAL units of the current coded picture, according to the dsci_hash_method_type specified in the preceding digitally signed content initialization SEI message.

[0391] Example 4

[0392] The use of a digitally signed content selection SEI message requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0393] The digitally signed content selection SEI message provides a mechanism for associating coded pictures with one of the verification substreams indicated in the digitally signed content initialization SEI message.

[0394] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content selection SEI message, the digitally signed content initialization SEI message must be placed in decoding order prior to the digitally signed content selection SEI message.

[0395] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content selection SEI message.

[0396] If a digitally signed content selection SEI message exists in any AU of CVS, said message must precede all non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList of said AU.

[0397] The verification substream ID (dscs_verification_substream_id) indicates the verification substream to which non-VCL NAL units with NAL unit type identifiers corresponding to any of the values ​​in nonVclDigitallySignedNalUnitsList and VCL NAL units of the current coded picture are assigned. If a digitally signed content initialization SEI message exists for the current coded video sequence but a digitally signed content selection SEI message does not exist for the coded picture, the value of dscs_verification_substream_id is inferred to be equal to 0. The value of dscs_verification_substream_id must be within the range (inclusive) from 0 to dsci_num_verification_substreams_minus1. The substreams indicated by the sequential dscs_verification_substream_id must consist of NAL units belonging to the same layer and / or temporal sublayer.

[0398] The message digest for the verification substream with id equal to dscs_verification_substream_id is updated using non-VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList and the VCL NAL units of the current coded picture, according to the dsci_hash_method_type specified in the preceding digitally signed content initialization SEI message.

[0399] Example 5

[0400] The use of digitally signed content verification SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0401] The digitally signed content verification SEI message provides a mechanism for verifying the digital signature of video content.

[0402] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content verification SEI message.

[0403] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content verification SEI message, the digitally signed content initialization SEI message must precede the digitally signed content verification SEI message. If the PU includes both a digitally signed content selection SEI message and a digitally signed content verification SEI message, the digitally signed content selection SEI message must precede the digitally signed content verification SEI message.

[0404] If a digitally signed content verification SEI message exists in the PU of the CVS, a non-VCL NAL unit or a VCL NAL unit having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList must not be assigned to the substream indicated by dscv_verification_substream_id unless one or more of the conditions are met, such as the bitstream being terminated, a new CVS being started, or a new digitally signed content initialization SEI message being received.

[0405] The verification substream ID (dscv_verification_substream_id) indicates the verification substream to which the above SEI message applies.

[0406] The value obtained by adding 1 to the signature length (dscv_signature_length_in_octets_minus1) (i.e., dscv_signature_length_in_octets_minus1+1) specifies the length of the syntax element dscv_signature in octets, where 1 octet consists of 8 bits.

[0407] The signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0408] The verification of a bitstream signature consists of the following sequential steps.

[0409] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: for a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 8 (composition of the identification string (IdString)).

[0410] (2) The reference message digest RefDigest is determined as follows.

[0411] If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1, and the condition is that said RefDigest has a layer id and temporal sublayer equal to or lower than the layer id and temporal sublayer of the substream with id equal to dscv_verification_substream_id. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to a digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id.

[0412] In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI message (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 8.

[0413] Otherwise, the reference message digest RefDigest is the last calculated message digest for the validation substream with id 0. (3) The identification string IdString is constructed by concatenating the binary representations of the reference message digest RefDigest, the current message digest, dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 8.

[0414] [Table 8]

[0415]

[0416] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0417] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0418] NOTE 1 ― Since the bitstring used for signature verification includes RefDigest, it is possible to verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList used in the calculation of the current message digest are correct, and further verify that non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the additional values ​​of nonVclDigitallySignedNalUnitsList were not added to the bitstream, and that non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList were not removed from the bitstream.

[0419] NOTE 2 ― When the decoder is tuned to the bitstream, the value of RefDigest cannot be calculated accurately, so the IdString configured for the first digitally signed content verification SEI message cannot be verified. However, starting from the second digitally signed content verification SEI message, the signatures can be verified. After verification, the message digest for the verification substream with the same ID as dscv_verification_substream_id is reinitialized for the specified dsci_hash_method_type according to the NIST FIPS PUB 180-4 specification.

[0420] Example 6

[0421] The use of digitally signed content verification SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0422] The digitally signed content verification SEI message provides a mechanism for verifying the digital signature of video content.

[0423] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content verification SEI message.

[0424] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content verification SEI message, the digitally signed content initialization SEI message must precede the digitally signed content verification SEI message.

[0425] If a PU contains both a digitally signed content selection SEI message and a digitally signed content verification SEI message, the digitally signed content selection SEI message must precede the digitally signed content verification SEI message. If a digitally signed content verification SEI message exists in a PU of a CVS, a non-VCL NAL unit or a VCL NAL unit having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList must not be assigned to the substream indicated by dscv_verification_substream_id unless one or more of the conditions are met, such as the bitstream being terminated, a new CVS being started, or a new digitally signed content initialization SEI message being received.

[0426] The verification substream ID (dscv_verification_substream_id) indicates the verification substream to which the above SEI message applies.

[0427] The value obtained by adding 1 to the signature length (dscv_signature_length_in_octets_minus1) (i.e., dscv_signature_length_in_octets_minus1+1) specifies the length of the syntax element dscv_signature in octets, where 1 octet consists of 8 bits.

[0428] The signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0429] The verification of a bitstream signature consists of the following sequential steps.

[0430] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 9 (composition of the identification string (IdString)).

[0431] (2) The reference message digest RefDigest is determined as follows. If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id. In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI messages (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 9. Otherwise, the reference message digest RefDigest is the last message digest calculated for the verification substream with ID 0.

[0432] (3) As shown in Table 9, the identification string IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of the reference message digest RefDigest and the current message digest.

[0433] [Table 9]

[0434]

[0435] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0436] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0437] (NOTE 1) — Since the bitstring used for signature verification includes RefDigest, it is possible to verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList used in the calculation of the current message digest are correct, and further verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the additional values ​​of nonVclDigitallySignedNalUnitsList were not added to the bitstream, and that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList were not removed from the bitstream.

[0438] (NOTE 2) ― When the decoder is tuned to the bitstream, the value of RefDigest cannot be calculated accurately, so the IdString configured for the first digitally signed content verification SEI message cannot be verified. However, starting from the second digitally signed content verification SEI message, the signatures can be verified.

[0439] (NOTE 3) ― To ensure temporal consistency and scalability, substreams can be configured to belong to the same layer and temporal sublayer.

[0440] After verification, the message digest for the verification substream with the same ID as dscv_verification_substream_id is reinitialized for the specified dsci_hash_method_type according to the NIST FIPS PUB 180-4 specification.

[0441] Example 7

[0442] The use of digitally signed content verification SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0443] The digitally signed content verification SEI message provides a mechanism for verifying the digital signature of video content.

[0444] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content verification SEI message.

[0445] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content verification SEI message, the digitally signed content initialization SEI message must precede the digitally signed content verification SEI message.

[0446] If a PU contains both a digitally signed content selection SEI message and a digitally signed content verification SEI message, the digitally signed content selection SEI message must precede the digitally signed content verification SEI message. If a digitally signed content verification SEI message exists in a PU of a CVS, a non-VCL NAL unit or a VCL NAL unit having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList must not be assigned to the substream indicated by dscv_verification_substream_id unless one or more of the conditions are met, such as the bitstream being terminated, a new CVS being started, or a new digitally signed content initialization SEI message being received.

[0447] The verification substream ID (dscv_verification_substream_id) indicates the verification substream to which the above SEI message applies.

[0448] The value obtained by adding 1 to the signature length (dscv_signature_length_in_octets_minus1) (i.e., dscv_signature_length_in_octets_minus1+1) specifies the length of the syntax element dscv_signature in octets, where 1 octet consists of 8 bits.

[0449] The signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0450] The verification of a bitstream signature consists of the following sequential steps.

[0451] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 10 (composition of the identification string (IdString)).

[0452] (2) The reference message digest RefDigest is determined as follows. If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id. In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI messages (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 10. Otherwise, the reference message digest RefDigest is the last message digest calculated for the verification substream with ID 0.

[0453] A verification substream substrA must not refer to another verification substream substrB that contains pictures belonging to a higher layer and / or a higher temporal sublayer than the pictures within substrA.

[0454] (3) The identification string IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 10.

[0455] [Table 10]

[0456]

[0457] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0458] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0459] (NOTE 1) — Since the bitstring used for signature verification includes RefDigest, it is possible to verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList used in the calculation of the current message digest are correct, and further verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the additional values ​​of nonVclDigitallySignedNalUnitsList were not added to the bitstream, and that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList were not removed from the bitstream.

[0460] (NOTE 2) ― When the decoder is tuned to the bitstream, the value of RefDigest cannot be calculated accurately, so the IdString configured for the first digitally signed content verification SEI message cannot be verified. However, starting from the second digitally signed content verification SEI message, the signatures can be verified.

[0461] After verification, the message digest for the verification substream with the same ID as dscv_verification_substream_id is reinitialized for the specified dsci_hash_method_type according to the NIST FIPS PUB 180-4 specification.

[0462] Example 8

[0463] The use of digitally signed content verification SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0464] The digitally signed content verification SEI message provides a mechanism for verifying the digital signature of video content.

[0465] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content verification SEI message.

[0466] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content verification SEI message, the digitally signed content initialization SEI message must precede the digitally signed content verification SEI message.

[0467] If a PU contains both a digitally signed content selection SEI message and a digitally signed content verification SEI message, the digitally signed content selection SEI message must precede the digitally signed content verification SEI message. If a digitally signed content verification SEI message exists in a PU of a CVS, a non-VCL NAL unit or a VCL NAL unit having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList must not be assigned to the substream indicated by dscv_verification_substream_id unless one or more of the conditions are met, such as the bitstream being terminated, a new CVS being started, or a new digitally signed content initialization SEI message being received.

[0468] The verification substream ID (dscv_verification_substream_id) indicates the verification substream to which the above SEI message applies.

[0469] The value obtained by adding 1 to the signature length (dscv_signature_length_in_octets_minus1) (i.e., dscv_signature_length_in_octets_minus1+1) specifies the length of the syntax element dscv_signature in octets, where 1 octet consists of 8 bits.

[0470] The signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0471] The verification of a bitstream signature consists of the following sequential steps.

[0472] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 11 (composition of the identification string (IdString)).

[0473] (2) The reference message digest RefDigest is determined as follows.

[0474] If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id.

[0475] In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI message (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 11.

[0476] Otherwise, the reference message digest RefDigest is the last calculated message digest for the validation substream with id 0.

[0477] A verification substream substrA must not refer to another verification substream substrB containing NAL units belonging to a higher layer and / or temporal sublayer than the NAL units within substrA.

[0478] (3) The identification string IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 11.

[0479] [Table 11]

[0480]

[0481] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0482] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0483] (NOTE 1) — Since the bitstring used for signature verification includes RefDigest, it is possible to verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList used in the calculation of the current message digest are correct, and further verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the additional values ​​of nonVclDigitallySignedNalUnitsList were not added to the bitstream, and that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList were not removed from the bitstream.

[0484] (NOTE 2) ― When the decoder is tuned to the bitstream, the value of RefDigest cannot be calculated accurately, so the IdString configured for the first digitally signed content verification SEI message cannot be verified. However, starting from the second digitally signed content verification SEI message, the signatures can be verified.

[0485] After verification, the message digest for the verification substream with the same ID as dscv_verification_substream_id is reinitialized for the specified dsci_hash_method_type according to the NIST FIPS PUB 180-4 specification.

[0486] Example 9

[0487] The use of digitally signed content verification SEI messages requires the following definition: nonVclDigitallySignedNalUnitsList, which is a list of non-VCL NAL unit type identifiers.

[0488] The digitally signed content verification SEI message provides a mechanism for verifying the digital signature of video content.

[0489] If a CVS does not include a digitally signed content initialization SEI message, the CLVS of that CVS must not include a digitally signed content verification SEI message.

[0490] If the AU includes both a digitally signed content initialization SEI message and a digitally signed content verification SEI message, the digitally signed content initialization SEI message must precede the digitally signed content verification SEI message.

[0491] If a PU contains both a digitally signed content selection SEI message and a digitally signed content verification SEI message, the digitally signed content selection SEI message must precede the digitally signed content verification SEI message. If a digitally signed content verification SEI message exists in a PU of a CVS, a non-VCL NAL unit or a VCL NAL unit having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList must not be assigned to the substream indicated by dscv_verification_substream_id unless one or more of the conditions are met, such as the bitstream being terminated, a new CVS being started, or a new digitally signed content initialization SEI message being received.

[0492] The verification substream ID (dscv_verification_substream_id) indicates the verification substream to which the above SEI message applies.

[0493] The value obtained by adding 1 to the signature length (dscv_signature_length_in_octets_minus1) (i.e., dscv_signature_length_in_octets_minus1+1) specifies the length of the syntax element dscv_signature in octets, where 1 octet consists of 8 bits.

[0494] The signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0495] The verification of a bitstream signature consists of the following sequential steps.

[0496] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 12 (composition of the identification string (IdString)).

[0497] (2) The reference message digest RefDigest is determined as follows. If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id. In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI messages (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 12. Otherwise, the reference message digest RefDigest is the last message digest calculated for the verification substream with ID 0.

[0498] RefDigest must not contain any NAL units with a nuh_layer_id and / or TemporalId higher than those used in CurrDigest.

[0499] (3) The identification string IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 12.

[0500] [Table 12]

[0501]

[0502] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0503] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0504] (NOTE 1) — Since the bitstring used for signature verification includes RefDigest, it is possible to verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList used in the calculation of the current message digest are correct, and further verify that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the additional values ​​of nonVclDigitallySignedNalUnitsList were not added to the bitstream, and that the non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList were not removed from the bitstream.

[0505] (NOTE 2) ― When the decoder is tuned to the bitstream, the value of RefDigest cannot be calculated accurately, so the IdString configured for the first digitally signed content verification SEI message cannot be verified. However, starting from the second digitally signed content verification SEI message, the signatures can be verified.

[0506] After verification, the message digest for the verification substream with the same ID as dscv_verification_substream_id is reinitialized for the specified dsci_hash_method_type according to the NIST FIPS PUB 180-4 specification.

[0507] Example 10

[0508] In the digitally signed content verification SEI message syntax, the signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0509] The verification of a bitstream signature consists of the following sequential steps.

[0510] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having NAL unit type identifiers corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 13 (composition of the identification string (IdString)).

[0511] (2) The reference message digest RefDigest is determined as follows. If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id. In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI message (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 13. Otherwise, the reference message digest RefDigest is the last message digest calculated for the verification substream with ID 0.

[0512] As a requirement for bitstream conformance, if the verification of a verification substream A with a dscv_verification_substream_id value greater than 0 uses a reference message digest from another verification substream B, all of the following conditions must be satisfied: The PUs associated with substream B must not belong to a layer in a higher hierarchy than the highest layer of the PUs associated with substream A. Additionally, the PUs associated with substream B must not belong to a temporal sublayer in a higher hierarchy than the highest temporal sublayer of the PUs associated with substream A.

[0513] (3) The identification string IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 13.

[0514] [Table 13]

[0515]

[0516] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0517] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0518] Example 11

[0519] In the digitally signed content verification SEI message syntax, the signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0520] The verification of a bitstream signature consists of the following sequential steps.

[0521] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 14.

[0522] (2) The reference message digest RefDigest is determined as follows. If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id. In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI message (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 14. Otherwise, the reference message digest RefDigest is the last message digest calculated for the verification substream with ID 0.

[0523] As a requirement for bitstream conformance, if the verification of a verification substream A with a dscv_verification_substream_id value greater than 0 uses a reference message digest from another verification substream B, all of the following conditions must be satisfied: NAL units associated with substream B must not have a nuh_layer_id higher than the highest nuh_layer_id of the NAL units associated with substream A. Additionally, NAL units associated with substream B must not have a TemporalId higher than the highest TemporalId of the NAL units associated with substream A.

[0524] (3) As shown in Table 14 (Composition of IdString), the IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of dsci_hash_method_type and (if present) dsci_content_uuid, as illustrated in Figure XXX.

[0525] [Table 14]

[0526]

[0527] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0528] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0529] Example 12

[0530] In the digitally signed content verification SEI message syntax, the signature (dscv_signature) contains a digital signature for the verification substream indicated by dscv_verification_substream_id.

[0531] The verification of a bitstream signature consists of the following sequential steps.

[0532] (1) The calculation of the message digest referred to as CurrDigest is finalized as follows: For a verification substream where id is the same as dscv_verification_substream_id, the concatenation of non-VCL NAL units and VCL NAL units having a NAL unit type identifier corresponding to any of the values ​​of nonVclDigitallySignedNalUnitsList is padded according to the specifications set forth in NIST FIPS PUB 180-4, and it is sufficient to pad only the last NAL unit of the verification substream. In addition, the calculation of the message digest CurrDigest is finalized according to the specifications of NIST FIPS PUB 180-4, and the length of the message digest (in bits) is given by Table 15 (composition of the identification string (IdString)).

[0533] (2) The reference message digest RefDigest is determined as follows. If dscv_verification_substream_id is greater than 0, the reference message digest RefDigest is the last calculated message digest for the verification substream with id equal to dscv_verification_substream_id-1. As a requirement for bitstream conformance, any digitally signed content verification SEI associated with a verification substream id equal to dscv_verification_substream_id-1 must exist prior to the digitally signed content verification SEI message having a verification substream id equal to dscv_verification_substream_id.

[0534] In addition, if the current digitally signed content verification SEI message is the first digitally signed content verification SEI with verification ID 0 in the coded video sequence, and the preceding coded video sequence did not contain any digitally signed content initialization SEI message (this includes cases where the current coded video sequence is the first coded video sequence in the bitstream), RefDigest is set to be the same as a bitstring with all DigestSize bits set to 1, where DigestSize is the size of the message digest specified in Table 15.

[0535] Otherwise, the reference message digest RefDigest is the last calculated message digest for the validation substream with id 0.

[0536] As a requirement for bitstream conformance, if the verification of a verification substream A with a dscv_verification_substream_id value greater than 0 uses a reference message digest from another verification substream B, all of the following conditions must be satisfied: The PUs associated with substream B must not belong to a layer in a higher hierarchy than the highest layer of the PUs associated with substream A. Additionally, the PUs associated with substream B must not belong to a temporal sublayer in a higher hierarchy than the highest temporal sublayer of the PUs associated with substream A.

[0537] As a requirement for bitstream conformance, if the verification of a verification substream A with a dscv_verification_substream_id value greater than 0 uses a reference message digest from another verification substream B, all of the following conditions must be satisfied: NAL units associated with substream B must not have a nuh_layer_id higher than the highest nuh_layer_id of the NAL units associated with substream A. Additionally, NAL units associated with substream B must not have a TemporalId higher than the highest TemporalId of the NAL units associated with substream A.

[0538] (3) The identification string IdString is constructed by concatenating the reference message digest RefDigest, the current message digest, and the binary representations of dsci_hash_method_type and (if present) dsci_content_uuid, as shown in Table 15.

[0539] [Table 15]

[0540]

[0541] The number of bits for RefDigest is determined by the value of dsci_hash_method_type that was valid when calculating the value of RefDigest, the number of bits for CurrDigest is determined by the current value of dsci_hash_method_type, the value of dsci_hash_method_type is represented as 8 bits, and (if present) the value of dsci_content_uuid is represented as 128 bits.

[0542] (4) The identification string IdString represents the message used to verify the signature. The signature verification algorithm and the public key used to verify the signature are indicated by the syntax elements dsci_use_key_register_idx_flag, dsci_key_source_uri, and dsci_key_register_idx when dsci_use_key_register_idx_flag is 1.

[0543] Example 13

[0544] When text is included in the VSEI: As a requirement for bitstream conformance, if the verification of a verification substream A with a dscv_verification_substream_id value greater than 0 uses a reference message digest from another verification substream B, all of the following conditions must be satisfied. The NAL units of the PUs associated with substream B must not belong to a layer in a higher hierarchy than the highest layer of the NAL units of the PUs associated with substream A. If the highest layers of the NAL units in substream A and B are the same, the NAL units of the PUs associated with substream B must not belong to a temporal sublayer in a higher hierarchy than the highest temporal sublayer of the NAL units of the PUs associated with substream A.

[0545] When text is included in the VVC interface: As a requirement for bitstream conformance, if the verification of a verification substream A with a dscv_verification_substream_id value greater than 0 uses a reference message digest from another verification substream B, all of the following conditions must be satisfied. The NAL units of the PUs associated with substream B must not have a nuh_layer_id higher than the highest nuh_layer_id of the NAL units of the PUs associated with substream A. If the highest nuh_layer_id of the NAL units in substream A and B is the same, the NAL units of the PUs associated with substream B must not have a TemporalId higher than the highest TemporalId of the NAL units of the PUs associated with substream A.

[0546] FIG. 28 illustrates a encoding method according to embodiments.

[0547] An encoding device performing an encoding method according to embodiments performs the source device of FIG. 1, an encoder of FIG. 2, a system of FIG. 4, encoding according to the method of FIG. 5 to FIG. 22, generation of SEI messages such as FIG. 23 to FIG. 26, a reference method for verifying the integrity of a Temporal ID-related substream of FIG. 27, and a method of FIG. 28.

[0548] The method according to the embodiments may include the step of encoding a picture (S2800); the step of generating a digitally signed content initialization SEI message (S2810); the step of generating a digitally signed content selection SEI message (S2820); and / or the step of generating a digitally signed content verification SEI message (S2830); etc.

[0549] The method of FIG. 28 is performed by a device, the device comprises a memory; and at least one processor connected to the memory; and the at least one processor is configured to: encode a picture; generate a Digitally Signed Content Initialization SEI message; generate a Digitally Signed Content Selection SEI message; and generate a Digitally Signed Content Verification SEI message.

[0550] The embodiments further include a computer-readable storage medium for storing a bitstream generated by the method according to FIG. 28.

[0551] The embodiments further include a method comprising the steps of: acquiring a bitstream; encoding a picture in the bitstream; generating a digitally signed content initialization SEI message; generating a digitally signed content selection SEI message; and generating a digitally signed content verification SEI message; and transmitting data including the bitstream.

[0552] The methods in Fig. 28 and Fig. 29 can correspond to each other as inverse processes.

[0553] FIG. 29 illustrates a decoding method according to embodiments.

[0554] A decoding device performing a decoding method according to embodiments performs the receiving device of FIG. 1, a decoder of FIG. 3, a system of FIG. 4, decoding according to the method of FIG. 5 to FIG. 22, acquisition of SEI messages such as FIG. 23 to FIG. 26, a reference method for verifying the integrity of a Temporal ID-related substream of FIG. 27, and the method of FIG. 29.

[0555] Referring together to FIG. 29, FIG. 24, FIG. 26, etc., the method according to the embodiments may include the step of obtaining a digitally signed content initialization SEI message from a bitstream (S2900); the step of obtaining a digitally signed content selection SEI message from a bitstream (S2910); the step of obtaining a digitally signed content verification SEI message from a bitstream (S2920); and / or the step of decoding a picture within the bitstream (S2930); etc.

[0556] Referring to the dscv_verification_substream_id, Temporal ID, etc. in Figs. 27 and 26, the Digitally Signed Content Initialization SEI message, the Digitally Signed Content Selection SEI message, and the Digitally Signed Content Verification SEI message are included in the Digitally Signed Content SEI messages (DSC); the Digitally Signed Content Initialization SEI message establishes the verification context of the Coded Video Sequence (CVS) within the bitstream; the Digitally Signed Content Selection SEI message allocates NAL units to the verification substream; and the Digitally Signed Content Verification SEI message is used to verify the integrity of the substream based on the current digest and the RefDigest, and the RefDigest is the verification substream included in the Digitally Signed Content Verification SEI message. Represents temporal consistency based on the hash value of the substream identified by the ID (dscv_verification_substream_id).

[0557] When referring together with Fig. 24 dsci_num_verification_substreams_minus1 and Fig. 27 hierarchical relationships, the digitally signed content initialization SEI message includes an element (dsci_num_verification_substreams_minus1) related to the number of substreams, the substreams are referenced based on hierarchical reference relationships, and each substream includes a picture for a temporal ID.

[0558] A bitstream according to the embodiments has constraints, the constraints being: substreams for a digitally signed content initialization SEI message belong to the same layer or the same temporal ID, a first substream having a first temporal ID cannot refer to a second substream having a second temporal ID higher than the first temporal ID, and a first substream cannot refer to a third substream having a third temporal ID lower than the first temporal ID;

[0559] A bitstream according to the embodiments has constraints, the constraints being: based on the case where the verification of a first substream having a value of a verification substream ID (dscv_verification_substream_id) greater than 0 in a digitally signed content verification SEI message uses the reference message digest of the second substream, the Network Abstraction Layer (NAL) unit of the Prediction Unit (PU) associated with the second substream does not belong to a layer higher than the highest layer among the NAL units of the PU associated with the first substream; and based on the fact that the highest layers of the NAL units in the first substream and the second substream are the same, the NAL unit of the PU associated with the second substream does not belong to a time sublayer of a layer higher than the highest time sublayer among the NAL units of the PU associated with the first substream. (It is a requirement of bitstream conformance that when the verification of a verification substream substreamA with dscv_verification_substream_id value greater than 0, uses reference message digest from a verification substream substreamB, both of the following shall be true: - NAL units of PUs that are associated with substreamB shall not belong to a layer that is in the higher hierarchy than the highest layer of the NAL units of PUs that are associated with substreamA.- When the highest layer of NAL units in substreamA and B is equal, NAL units PUs that are associated with substreamB shall not belong to temporal sub layer that is higher hierarchy than the highest temporal sub layer of the NAL units of PUs that are associated with substreamA).

[0560] A bitstream according to the embodiments has constraints, the constraints being: based on the case where the verification of a first substream having a value of a verification substream ID (dscv_verification_substream_id) greater than 0 in a digitally signed content verification SEI message uses the reference message digest of the second substream, the layer ID (nuh_layer_id) of the Network Abstraction Layer (NAL) unit of the Prediction Unit (PU) associated with the second substream is not higher than the top layer ID (nuh_layer_id) of the NAL unit of the PU associated with the first substream; and based on the fact that the top layer IDs of the NAL units in the first substream and the second substream are the same, the temporal ID (TemporalId) of the NAL unit of the PU associated with the second substream is not higher than the top temporal ID (TemporalId) of the NAL unit of the PU associated with the first substream; (It is a requirement of bitstream conformance that when the verification of a verification substream substreamA with dscv_verification_substream_id value greater than 0, uses reference message digest from a verification substream substreamB, both of the following shall be true: - NAL units of PUs that are associated with substreamB shall not have nuh_layer_id that is higher than the highest nuh_layer_id of NAL units of PUs that are associated with substreamA.- When the highest nuh_layer_id of NAL units in substreamA and B is equal, NAL units of PUs that are associated with substreamB shall not have TemporalId that is higher than the highest TemporalId of NAL units of PUs that are associated with substreamA).

[0561] The method of FIG. 29 is performed by a device, the device comprises a memory; and at least one processor connected to the memory; and the at least one processor is configured to: obtain a digitally signed content initialization SEI message from a bitstream; obtain a digitally signed content selection SEI message from a bitstream; obtain a digitally signed content verification SEI message from a bitstream; and decode a picture in the bitstream.

[0562] The method and apparatus according to the embodiments provide the following technical effects.

[0563] As a bitstream conformance requirement, constraints are specified to ensure that each substream within the initialization unit (i.e., the interval before another Initialization SEI message appears) consists only of NAL units belonging to the same layer, and to prevent a substream from referencing another substream containing information from a higher layer. Furthermore, by limiting the targets a substream can reference to other substreams containing information from the same or lower layer, it is structurally possible to block lower layers from using information from upper layers during the verification or integrity check process. Consequently, verification inconsistencies, unpredictable dependencies, and the possibility of authentication bypass through malicious manipulation that may arise from cross-layer reverse referencing or circular referencing are reduced, and security and reliability are enhanced as verification procedures for each substream are performed consistently according to layer relationships. Furthermore, as consistency in the layer configuration within the initialization unit is guaranteed, the complexity of the decoder / verifier implementation is alleviated, and the interoperability between various receiving devices and the reproducibility of conformity determination are improved.

[0564] The embodiments have been described in terms of methods and / or devices, and the description of the methods and the description of the devices may be applied complementarily.

[0565] Although the drawings have been described separately for the convenience of explanation, it is also possible to design a new embodiment by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by a person skilled in the art, falls within the scope of the claims of the embodiments. The apparatus and method according to the embodiments are not limited to the configuration and method of the embodiments described above; rather, the embodiments may be configured by selectively combining all or part of each embodiment to allow for various modifications. Although preferred embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above. It is not only possible for a person skilled in the art to make various modifications without departing from the essence of the embodiments claimed in the claims, but such modifications should not be understood individually from the technical concept or perspective of the embodiments.

[0566] Various components of the device of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented as a single chip, for example, a single hardware circuit. Depending on the embodiments, the components according to the embodiments may each be implemented as separate chips. Depending on the embodiments, at least one of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and one or more programs may include instructions for performing or executing any one or more of the operations / methods according to the embodiments. Executable instructions for performing the methods / operations of the device according to the embodiments may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. Additionally, memory according to the embodiments may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. In addition, it may also include implementation in the form of carrier waves, such as transmission over the Internet. Furthermore, processor-readable recording media are distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0567] In this document, " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Additionally, "A / B / C" means "at least one of A, B and / or C." Also, "A, B, C" means "at least one of A, B and / or C." Additionally, in this document, "or" is interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." In other words, "or" in this document may mean "additionally or alternatively."

[0568] Terms such as "first," "second," etc., may be used to describe various components of the embodiments. However, the interpretation of the various components according to the embodiments should not be limited by these terms. These terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not imply the same user input signals unless clearly indicated in the context.

[0569] The terms used to describe the embodiments are intended for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless explicitly indicated in the context. Expressions of and / or are used to mean including all possible combinations between the terms. Expressions of include describe the presence of features, numbers, steps, elements, and / or components and do not imply the exclusion of additional features, numbers, steps, elements, and / or components. Conditional expressions such as "if" or "when" used to describe the embodiments are not limited to being optional. It is intended to be interpreted as "when a specific condition is satisfied," "when a related action is performed in response to a specific condition," or "when a related definition is interpreted."

[0570] Additionally, operations according to the embodiments described herein may be performed by a transmitting and receiving device including memory and / or a processor, depending on the embodiments. The memory may store programs for processing / controlling operations according to the embodiments, and the processor may control various operations described in this document. The processor may be referred to as a controller, etc. Operations in the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in the processor or in memory.

[0571] Meanwhile, the operation according to the embodiments described above may be performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting and receiving device may include a transmitting and receiving unit for transmitting and receiving media data, a memory for storing instructions (program code, algorithm, flowchart and / or data) for a process according to the embodiments, and a processor for controlling the operations of the transmitting and receiving devices.

[0572] The processor may be referred to as a controller, etc., and may correspond, for example, to hardware, software, and / or a combination thereof. The operation according to the embodiments described above may be performed by the processor. Additionally, the processor may be implemented as an encoder / decoder, etc., for the operation of the embodiments described above.

[0573] As described above, the relevant details have been explained in the best mode for carrying out the embodiments.

[0574] As described above, the embodiments may be applied wholly or partially to an image encoding method, an image encoding device, an image decoding method, an image decoding device, and a system.

[0575] Those skilled in the art may make various changes or modifications to the embodiments within the scope of the embodiments.

[0576] The embodiments may include modifications / variations, and such modifications / variations do not exceed the scope of the claims and their equivalents.

Claims

1. A step of obtaining a digitally signed content initialization SEI message from a bitstream; A step of obtaining a digitally signed content selection SEI message from the bitstream; A step of obtaining a digitally signed content verification SEI message from the bitstream above; and A step of decoding a picture within the bitstream; comprising method.

2. In Paragraph 1, The above digitally signed content initialization SEI message, the above digitally signed content selection SEI message, and the above digitally signed content verification SEI message are included in the digitally signed content SEI messages (DSC) SEI message, and The above Digitally Signed Content Initialization (SES) message establishes a verification context for a Coded Video Sequence (CVS) within the bitstream, and The above digitally signed content selection SEI message allocates NAL units to the verification substream, and The above digitally signed content verification SEI message is used to verify the integrity of the substream based on the current digest and the reference digest (RefDigest), and The above reference digest indicates temporal consistency based on the hash value of the substream identified based on the verification substream ID (dscv_verification_substream_id) included in the digitally signed content verification SEI message, method.

3. In Paragraph 1, The above digitally signed content initialization SEI message includes an element (dsci_num_verification_substreams_minus1) related to the number of substreams, and The above substreams are referenced based on hierarchical reference relationships, and including a picture for the temporal ID included in each substream, method.

4. In Paragraph 1, The above bitstream has constraints, The above constraints are: The substreams for the above digitally signed content initialization SEI message belong to the same layer or the same temporal ID, and A first substream having a first temporal ID cannot refer to a second substream having a second temporal ID higher than the first temporal ID, and The first substream cannot refer to a third substream having a third temporal ID lower than the first temporal ID; comprising method.

5. In Paragraph 1, The above bitstream has constraints, The above constraints are: Based on the case where the verification of a first substream having a value of a verification substream ID (dscv_verification_substream_id) greater than 0 within the above digitally signed content verification SEI message uses the reference message digest of the second substream, The Network Abstraction Layer (NAL) unit of the Prediction Unit (PU) associated with the second substream does not belong to a layer higher than the highest layer among the NAL units of the PU associated with the first substream; Based on the fact that the uppermost layer of the NAL units within the first substream and the second substream is the same, the NAL unit of the PU associated with the second substream does not belong to a time sublayer of a layer higher than the uppermost time sublayer among the NAL units of the PU associated with the first substream; comprising, method.

6. In Paragraph 1, The above bitstream has constraints, The above constraints are: Based on the case where the verification of a first substream having a value of a verification substream ID (dscv_verification_substream_id) greater than 0 within the above digitally signed content verification SEI message uses the reference message digest of the second substream, The layer ID (nuh_layer_id) of the NAL (Network Abstraction Layer) unit of the PU (Prediction Unit) associated with the second substream is not higher than the top layer ID (nuh_layer_id) of the NAL unit of the PU associated with the first substream; Based on the fact that the top layer IDs of the NAL units within the first substream and the second substream are identical, the Temporal ID of the NAL unit of the PU associated with the second substream is not higher than the top Temporal ID of the NAL unit of the PU associated with the first substream; comprising method.

7. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Obtain a digitally signed content initialization SEI message from the bitstream; Obtain a digitally signed content selection SEI message from the above bitstream; From the above bitstream, obtain a digitally signed content verification SEI message; and Configured to decode a picture within the above bitstream, device.

8. Step of encoding the picture; Step of generating a digitally signed content initialization (SEI) message; Step of generating a digitally signed content selection SEI message; and A step of generating a digitally signed content verification SEI message; comprising, method.

9. In Paragraph 8, The above digitally signed content initialization SEI message, the above digitally signed content selection SEI message, and the above digitally signed content verification SEI message are included in the digitally signed content SEI messages (DSC) SEI message, and The above Digitally Signed Content Initialization (SES) message establishes a verification context for a Coded Video Sequence (CVS) within the bitstream, and The above digitally signed content selection SEI message allocates NAL units to the verification substream, and The above digitally signed content verification SEI message is used to verify the integrity of the substream based on the current digest and the reference digest (RefDigest), and The above reference digest indicates temporal consistency based on the hash value of the substream identified based on the verification substream ID (dscv_verification_substream_id) included in the digitally signed content verification SEI message, method.

10. In Paragraph 8, The above digitally signed content initialization SEI message includes an element (dsci_num_verification_substreams_minus1) related to the number of substreams, and The above substreams are referenced based on hierarchical reference relationships, and including a picture for the temporal ID included in each substream, method.

11. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Encode the picture; Generate a Digitally Signed Content Initialization (SEI) message; Generate a digitally signed content selection SEI message; and Configured to generate a digitally signed content verification SEI message, device.

12. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 8.

13. Step of acquiring the bitstream, The above bitstream is generated based on the steps of: encoding a picture; generating a digitally signed content initialization SEI message; generating a digitally signed content selection SEI message; and generating a digitally signed content verification SEI message; and A method comprising the step of transmitting data including the bitstream above.

Images