Method for decoding image information, method for encoding image information, method for bitstream, and computer-readable storage medium

Abstract

A method according to the present disclosure comprises: acquiring a sei processing order (SPO) supplemental enhancement information (SEI) message, a processing order nesting (PON) SEI message, and a quality metrics (QM) SEI message; deriving a processing order of at least one SEI message on the basis of the SPO SEI message; acquiring at least one PON nested SEI message on the basis of the PON SEI message; and acquiring quality metrics on the basis of the QM SEI message, wherein, on the basis that the QM SEI message is a part of a processing chain of the SPO SEI message and is a PON nested SEI message, a value of quality metrics-related information included in the QM SEI message is the same as a value of quality metrics-related information included in a first QM SEI message which is a part of the processing chain of the SPO SEI message and is a PON nested SEI message.

Description

Method for decoding image information, method for encoding image information, method relating to a bitstream and a computer-readable storage medium

[0001] The present disclosure relates to a method for decoding image information, a method for encoding image information, a method for bitstreams, and a computer-readable storage medium.

[0002] Recently, the demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition), has been increasing across various fields. As video data becomes higher in resolution and quality, the relative amount of information or bits transmitted increases compared to conventional video data. This increase in transmitted information or bits leads to higher transmission and storage costs.

[0003] Accordingly, high-efficiency video compression technology is required to effectively transmit, store, and play back high-resolution, high-quality video information.

[0004] The present disclosure aims to improve the reliability of a coding system including an encoding device and a decoding device.

[0005] The present disclosure aims to improve the coding efficiency of a coding system including an encoding device and a decoding device.

[0006] The present disclosure aims to improve the data transmission efficiency of a coding system including an encoding device and a decoding device.

[0007] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0008] A method for decoding image information according to one aspect of the present disclosure comprises: acquiring an SPO (sei processing order) SEI (Supplemental Enhancement Information) message, a PON (Processing order nesting) SEI message, and a QM (Quality Metrics) SEI message; deriving a processing order of at least one SEI message based on the SPO SEI message; acquiring at least one PON nested SEI message based on the PON SEI message; and acquiring quality metrics based on the QM SEI message, wherein, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is a PON nested SEI message, the value of the quality metric information included in the QM SEI message is the same as the value of the quality metric information included in the first QM SEI message, which is part of the processing chain of the SPO SEI message and is a PON nested SEI message.

[0009] According to one aspect of the present disclosure, an apparatus for decoding image information comprises a memory and at least one processor connected to the memory, wherein the at least one processor acquires an SPO (sei processing order) SEI (Supplemental Enhancement Information) message, a PON (Processing order nesting) SEI message, and a QM (Quality Metrics) SEI message; derives a processing order of at least one SEI message based on the SPO SEI message; acquires at least one PON nested SEI message based on the PON SEI message; and acquires quality metrics based on the QM SEI message, wherein the value of quality metric information included in the QM SEI message is the same as the value of quality metric information included in the first QM SEI message, which is the PON nested SEI message and is part of the processing chain of the SPO SEI message.

[0010] In a method or device for decoding the above image information, based on the fact that the QM SEI message is the first QM SEI message which is a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating that the quality indicator related information exists.

[0011] In a method or device for decoding the above image information, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is not a PON-nested SEI message, the value of the quality indicator related information of the QM SEI message may be the same as the value of the quality indicator related information of the first QM SEI message that is part of the processing chain of the SPO SEI message and is not a PON-nested SEI message.

[0012] In a method or device for decoding the above image information, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is the first QM SEI message that is not a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating that the quality indicator related information exists.

[0013] In a method or device for decoding the above image information, the quality indicator related information may be any one of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality overall reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information.

[0014] A method for encoding image information according to one aspect of the present disclosure comprises: generating a Quality Metrics (QM) SEI (Supplemental Enhancement Information) message based on quality metrics; generating a Processing Order Nesting (PON) SEI message based on at least one Processing Order Nesting (PON) SEI message; generating a Sei Processing Order (SPO) SEI message based on the processing order of at least one SEI message including the at least one Processing Order Nesting (PON) SEI message and the QM SEI message; and encoding the QM SEI message, the PON SEI message, and the SPO SEI message, wherein, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is a PON-nested SEI message, the value of the quality metric information included in the QM SEI message is the same as the value of the quality metric information included in the first QM SEI message, which is part of the processing chain of the SPO SEI message and is a PON-nested SEI message.

[0015] According to one aspect of the present disclosure, an apparatus for encoding image information comprises a memory and at least one processor connected to the memory, wherein the at least one processor generates a QM (Quality Metrics) SEI (Supplemental Enhancement Information) message based on quality metrics; generates a PON (Processing Order Nesting) SEI message based on at least one PON-nested SEI message; generates an SPO (Sei Processing Order) SEI message based on the processing order of at least one SEI message including the at least one PON-nested SEI message and the QM SEI message; and encodes the QM SEI message, the PON SEI message, and the SPO SEI message, wherein, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is a PON-nested SEI message, the value of the quality metric-related information included in the QM SEI message is the same as the value of the quality metric-related information included in the first QM SEI message, which is part of the processing chain of the SPO SEI message and is a PON-nested SEI message.

[0016] In a method or device for encoding the above-mentioned image information, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is the first QM SEI message that is a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating that the quality indicator related information exists.

[0017] In a method or device for encoding the above image information, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is not a PON-nested SEI message, the value of the quality indicator related information of the QM SEI message may be the same as the value of the quality indicator related information of the first QM SEI message that is part of the processing chain of the SPO SEI message and is not a PON-nested SEI message.

[0018] In a method or device for encoding the above-mentioned image information, based on the fact that the QM SEI message is part of the processing chain of the SPO SEI message and is the first QM SEI message that is not a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating that the quality indicator-related information exists.

[0019] In a method or device for encoding the above image information, the quality indicator related information may be any one of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information.

[0020] A method for a bitstream according to one aspect of the present disclosure comprises generating said bitstream and transmitting said bitstream, wherein said bitstream is generated based on generating a QM (Quality Metrics) SEI (Supplemental Enhancement Information) message based on quality metric information, generating a PON (Processing Order Nesting) SEI message based on at least one PON-nested SEI message, generating an SPO (Sei Processing Order) SEI message based on processing order information of at least one SEI message including said at least one PON-nested SEI message and said QM SEI message, and generating said bitstream based on encoding said QM SEI message, said PON SEI message and said SPO SEI message, and wherein said QM SEI message is part of the processing chain of said SPO SEI message and is a PON-nested SEI message, the value of the quality metric information included in said QM SEI message is part of the processing chain of said SPO SEI message and is a PON-nested SEI message, and the value of the quality metric information included in said QM SEI message is part of the processing chain of said SPO SEI message and is a PON-nested SEI message and It is the same.

[0021] According to one aspect of the present disclosure, an apparatus for a bitstream comprises at least one processor for generating the bitstream and a transmission unit for transmitting the bitstream, wherein the bitstream is generated based on generating a QM (Quality Metrics) SEI (Supplemental Enhancement Information) message based on quality metric information, generating a PON (Processing Order Nesting) SEI message based on at least one PON-nested SEI message, generating an SPO (Sei Processing Order) SEI message based on processing order information of at least one SEI message including the at least one PON-nested SEI message and the QM SEI message, and generating based on encoding the QM SEI message, the PON SEI message, and the SPO SEI message, wherein the value of the quality metric related information included in the QM SEI message is part of the processing chain of the SPO SEI message and is a PON-nested SEI message, the value of the quality metric related information included in the first QM SEI message is part of the processing chain of the SPO SEI message and is a PON-nested SEI message. It is the same as the value.

[0022] According to one aspect of the present disclosure, in a computer-readable storage medium, the storage medium stores a bitstream, and the bitstream generates a QM (Quality Metrics) SEI (Supplemental Enhancement Information) message based on quality metric information, generates a PON (Processing Order Nesting) SEI message based on at least one PON-nested SEI message, generates an SPO (Sei Processing Order) SEI message based on processing order information of at least one SEI message including the at least one PON-nested SEI message and the QM SEI message, and generates based on encoding the QM SEI message, the PON SEI message and the SPO SEI message, and based on the QM SEI message being part of the processing chain of the SPO SEI message and being a PON-nested SEI message, the value of the quality metric information included in the QM SEI message is the same as the value of the quality metric information included in the first QM SEI message being part of the processing chain of the SPO SEI message and being a PON-nested SEI message.

[0023] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0024] According to the present disclosure, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0025] According to the present disclosure, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0026] According to the present disclosure, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0027] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0028] FIG. 1 is a schematic diagram illustrating a video coding system to which an embodiment according to the present disclosure can be applied.

[0029] FIG. 2 is a schematic diagram showing an encoding device to which an embodiment according to the present disclosure can be applied.

[0030] FIG. 3 is a schematic diagram showing a decoding device to which an embodiment according to the present disclosure can be applied.

[0031] Figure 4 illustrates an exemplary hierarchical structure for a coded video / image.

[0032] FIG. 5 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0033] FIG. 6 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0034] FIG. 7 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0035] Hereinafter, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0036] In describing the embodiments of the present disclosure, detailed descriptions of known configurations or functions are omitted if it is determined that such descriptions could obscure the essence of the present disclosure. Furthermore, parts of the drawings unrelated to the description of the present disclosure have been omitted, and similar parts are denoted by similar reference numerals.

[0037] In the present disclosure, when a component is described as being "connected," "combined," or "joined" with another component, this may include not only a direct connection but also an indirect connection in which another component exists in between. Furthermore, when a component is described as "comprising" or "having" another component, this means that, unless specifically stated otherwise, it does not exclude the other component but may include an additional component.

[0038] In the present disclosure, terms such as first, second, etc. are used solely for the purpose of distinguishing one component from another and do not limit the order or importance of the components unless specifically stated otherwise. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and likewise, a second component in one embodiment may be referred to as a first component in another embodiment.

[0039] In this disclosure, distinct components are intended to clearly describe their respective features and do not imply that the components are separate. That is, multiple components may be integrated to form a single hardware or software unit, or a single component may be distributed to form multiple hardware or software units. Accordingly, such integrated or distributed embodiments are included within the scope of this disclosure, unless otherwise noted.

[0040] In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Furthermore, embodiments including additional components in addition to the components described in various embodiments are also included within the scope of the present disclosure.

[0041] The present disclosure relates to the encoding and decoding of images. For example, the methods and embodiments disclosed in this document may be applied to methods disclosed in the VVC (versatile video coding) standard, EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard) or next-generation video / image coding standards (e.g., H.267 or H.268).

[0042] The present disclosure presents various embodiments relating to video / image coding, and unless otherwise stated, said embodiments may be performed in combination with one another.

[0043] Unless newly defined in this disclosure, the terms used herein may have the ordinary meanings commonly used in the technical field to which this disclosure belongs.

[0044] In this disclosure, "video" may refer to a set of images over time. In this disclosure, "picture" generally refers to a unit representing a single image at a specific time, and a slice / tile is a unit that constitutes a part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may represent a rectangular area of ​​rows of CTUs within a tile in a picture. In this document, tile groups and slices may be used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0045] In the present disclosure, "pixel" or "pel" may refer to the smallest unit constituting a picture (or image). Additionally, "sample" may be used as a term corresponding to pixel. A sample may generally represent a pixel or a pixel value, may represent only the pixel / pixel value of the luminance component, or may represent only the pixel / pixel value of the chroma component.

[0046] In this disclosure, "unit" may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. A unit may include one luminance block and two chroma (e.g., cb, cr) blocks. Depending on the case, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0047] In the present disclosure, "current block" may mean one of "current coding block," "current coding unit," "block to be encoded," "block to be decoded," or "block to be processed." When prediction is performed, "current block" may mean "current prediction block" or "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, "current block" may mean "current transformation block" or "block to be transformed." When filtering is performed, "current block" may mean "block to be filtered."

[0048] In the present disclosure, "current block" may mean a block comprising both a luminous component block and a chroma component block, or "luma block of the current block," unless explicitly stated as a chroma block. The luminous component block of the current block may be expressed by including an explicit description of a luminous component block, such as "luma block" or "current luminous block." Additionally, the chroma component block of the current block may be expressed by including an explicit description of a chroma component block, such as "chroma block" or "current chroma block."

[0049] In the present disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Additionally, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or C."

[0050] In the present disclosure, "or" may be interpreted as "and / or". For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B". Alternatively, in the present disclosure, "or" may mean "additionally or alternatively".

[0051] FIG. 1 is a schematic diagram illustrating a video / image coding system to which an embodiment according to the present disclosure can be applied.

[0052] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image or data in the form of a file or streaming to the receiving device via a digital storage medium or a network.

[0053] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be called a video / image encoding device, and the decoding device may be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be composed of a separate device or an external component.

[0054] A video source may acquire video / images through processes such as video / image capture, synthesis, or generation. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, etc., and may generate video / images (electronically). For example, virtual video / images may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process in which related data is generated.

[0055] The encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0056] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0057] The decoding device can decode video / images by performing a series of procedures such as inverse quantization, inverse transform, and prediction corresponding to the operation of the encoding device.

[0058] The renderer can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0059] FIG. 2 is a schematic diagram illustrating an encoding device to which an embodiment according to the present disclosure can be applied.

[0060] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-predictor (221) and an intra-predictor (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit (250) may be referred to as a reconstructor or a reconstructed block generator. The above-described image segmentation unit (210), prediction unit (220), residual processing unit (230), entropy encoding unit (240), addition unit (250), and filtering unit (260) may be configured by one or more hardware components (e.g., an encoder chipset or processor) according to the embodiment. Additionally, the memory (270) may include a DPB (Decoded Picture Buffer) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0061] The image segmentation unit (210) can divide an input image (or picture, frame) input to an encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). A coding unit may be recursively divided into a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, a single coding unit may be divided into multiple coding units of a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For example, a quad-tree structure may be applied first, and a binary-tree structure and / or a ternary-tree structure may be applied later. Alternatively, a binary-tree structure may be applied first. A coding procedure according to the present disclosure may be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the maximum coding unit may be recursively divided into lower-depth coding units so that a coding unit of the optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may each be divided or partitioned from the final coding unit.The above prediction unit may be a unit of sample prediction, and the above transformation unit may be a unit that derives transformation coefficients and / or a unit that derives a residual signal from transformation coefficients.

[0062] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.

[0063] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, as illustrated, the unit that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231). The prediction unit (220) can perform a prediction for a block to be processed (hereinafter, current block) and generate a predicted block (predicted block) containing prediction samples for said current block. The prediction unit (220) can determine whether intra prediction is applied or inter prediction is applied in units of the current block or CU. The prediction unit (220) can generate various information regarding prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (240). The information regarding prediction can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0064] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a Planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0065] The inter prediction unit (221) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different from each other. The temporal neighboring blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc. A reference picture containing the aforementioned temporal surrounding blocks may be called a collocated picture (colPic). For example, the inter prediction unit (221) may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of surrounding blocks as motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0066] The prediction unit (220) may generate a prediction signal based on various prediction methods and / or prediction techniques described below. For example, the prediction unit (220) may apply intra prediction or inter prediction for the prediction of the current block, as well as apply intra prediction and inter prediction simultaneously. A prediction method that applies intra prediction and inter prediction simultaneously for the prediction of the current block may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit (220) may be based on an intra block copy (IBC) prediction mode or a palette mode for the prediction of the block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intra-coding or intra-prediction. When palette mode is applied, sample values ​​within a picture can be signaled based on information regarding palette tables and palette indices.

[0067] The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or to generate a residual signal. The subtraction unit (231) can generate a residual signal (residual signal, residual block, residual sample array) by subtracting the prediction signal (predicted block, prediction sample array) output from the prediction unit (220) from the input image signal (original block, original sample array). The generated residual signal can be transmitted to the conversion unit (232).

[0068] The transformation unit (232) can generate transform coefficients by applying a transformation technique to a residual signal. For example, the transformation technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transformation obtained based on a prediction signal generated using all previously reconstructed pixels. The transformation process may be applied to a block of pixels of the same size in a square, or to a block of variable size that is not square.

[0069] The quantization unit (233) can quantize the transformation coefficients and transmit them to the entropy encoding unit (240). The entropy encoding unit (240) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (233) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients.

[0070] The entropy encoding unit (240) can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (190) may encode information required for video / image restoration (e.g., values ​​of syntax elements) together or separately, in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The signaling information, transmitted information, and / or syntax elements mentioned in the present disclosure may be included in the video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream.

[0071] The above bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) for transmitting a signal output from the entropy encoding unit (240) and / or a storage unit (not shown) for storing it may be provided as an internal / external element of the encoding device (200), or the transmission unit may be provided as a component of the entropy encoding unit (240).

[0072] The quantized transformation coefficients output from the quantization unit (233) can be used to generate a residual signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit (234) and the inverse transformation unit (235).

[0073] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0074] The adder (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (250) may be called a reconstructed unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after undergoing filtering as described below.

[0075] The filtering unit (260) can improve subjective / objective quality by applying filtering to the restored signal. For example, the filtering unit (260) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (270), specifically in the DPB of memory (170). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240), as described below in the description of each filtering method. The information regarding filtering can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0076] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter-prediction unit (221). Through this, the encoding device (200) can avoid prediction mismatches between the encoding device (200) and the decoding device when inter-prediction is applied, and can also improve encoding efficiency.

[0077] The DPB in memory (270) can store a modified restored picture to be used as a reference picture in the inter prediction unit (221). Memory (270) can store motion information of blocks from which motion information is derived (or encoded) in the current picture and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter prediction unit (221) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory (270) can store restoration samples of restored blocks in the current picture and transmit them to the intra prediction unit (222).

[0078] FIG. 3 is a schematic diagram illustrating a decoding device to which an embodiment according to the present disclosure can be applied.

[0079] As illustrated in FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-predictor (332) and an intra-predictor (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). The aforementioned entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) may be configured by a single hardware component (e.g., a decoder chipset or a processor) according to an embodiment. Additionally, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0080] When a bitstream containing video / image information is input, the decoding device (300) can restore the image by performing a process corresponding to the process performed by the encoding device (200) of FIG. 2. For example, the decoding device (300) can perform decoding using a processing unit applied in the encoding device (200). Thus, the processing unit for decoding may be, for example, a coding unit. The coding unit may be a coding tree unit, or a maximum coding unit may be obtained by dividing it according to a quad tree structure, a binary tree structure, and / or a binary tree structure. And, the restored image signal decoded and output through the decoding device (300) can be played back through a playback device (not shown).

[0081] The decoding device (300) can receive a signal output from the encoding device (200) of FIG. 2 in the form of a bitstream. The received signal can be decoded through an entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The decoding device (300) can decode the picture based on the information regarding the parameter sets and / or the general constraint information. The signaling / received information and / or syntax elements described below can be obtained from the bitstream by decoding through the decoding procedure. For example, the entropy decoding unit (310) can decode information within the bitstream based on coding methods such as exponential coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model using information on the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information on symbols / bins decoded in the previous step, predict the probability of occurrence of the bin according to the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin after determining the context model. Among the information decoded in the entropy decoding unit (310), information regarding prediction is provided to the prediction unit (330), and residual values ​​for which entropy decoding was performed in the entropy decoding unit (310), i.e., quantized transformation coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive residual signals (residual blocks, residual samples, residual sample array). Additionally, among the information decoded in the entropy decoding unit (310), information regarding filtering can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of the entropy decoding unit (310). Meanwhile, the decoding device according to the present document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit (310), and the sample decoder may include at least one of the inverse quantization unit (321), inverse transform unit (322), adder (340), filtering unit (350), memory (360), inter prediction unit (332), and intra prediction unit (331).

[0082] In the inverse quantization unit (321), the quantized transformation coefficients can be inversely quantized to output transformation coefficients. The inverse quantization unit (321) can rearrange the quantized transformation coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device (200). The inverse quantization unit (321) can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain transformation coefficients.

[0083] In the inverse conversion unit (322), the conversion coefficients can be inversely converted to obtain a residual signal (residual block, residual sample array).

[0084] The prediction unit (330) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, information regarding the palette table and palette index can be included in the above video / image information and signaled.

[0085] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The description of the intra prediction unit (222) may be applied equally to the intra prediction unit (331). The referenced samples may be located in the neighborhood of the current block or located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit (331) may determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0086] The inter prediction unit (332) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) may construct a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-prediction can be performed based on various prediction modes (techniques), and information regarding the prediction may include information indicating the mode (technique) of inter-prediction for the current block.

[0087] The adder (340) can generate a restoration signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the prediction signal (predicted block, predicted sample array) output from the prediction unit (330) (including the inter prediction unit (332) and / or intra prediction unit (331)). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the restoration block. The description of the adder (250) can be applied equally to the adder (340). The adder (340) may be called a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after undergoing filtering as described below.

[0088] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0089] The filtering unit (350) can improve subjective / objective quality by applying filtering to the restored signal. For example, the filtering unit (350) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (360), specifically in the DPB of memory (360). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0090] The (modified) restored picture stored in the DPB of the memory (360) can be used as a reference picture in the inter-prediction unit (332). The memory (360) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (332) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (360) can store restoration samples of blocks restored within the current picture and transmit them to the intra-prediction unit (331).

[0091] In this specification, the embodiments described in the filtering unit (260), inter prediction unit (221), and intra prediction unit (222) of the encoding device (200) may be applied to the filtering unit (350), inter prediction unit (332), and intra prediction unit (331) of the decoding device (300) in the same or corresponding manner.

[0092] Figure 4 illustrates an exemplary hierarchical structure for a coded video / image.

[0093] Referring to Figure 4, the coded image is divided into a Video Coding Layer (VCL) that handles the decoding processing of the image and the image itself, a subsystem that transmits and stores the encoded information, and a Network Abstraction Layer (NAL) that exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0094] In VCL, VCL data containing compressed image data (slice data) can be generated, or parameter sets containing information such as Picture Parameter Set (PPS), Sequence Parameter Set (SPS), and Video Parameter Set (VPS), or SEI (Supplemental Enhancement Information) messages that are additionally required in the decoding process of the image can be generated.

[0095] In NAL, a NAL unit can be created by adding header information (NAL unit header) to the Raw Byte Sequence Payload (RBSP) generated in VCL. In this case, the RBSP refers to slice data, parameter sets, SEI messages, etc. generated in VCL. The NAL unit header may include NAL unit type information specified according to the RBSP data included in the NAL unit.

[0096] As shown in FIG. 4, NAL units can be classified into VCL NAL units and Non-VCL NAL units depending on the RBSP generated in VCL. A VCL NAL unit may refer to a NAL unit containing information about an image (slice data), and a Non-VCL NAL unit may refer to a NAL unit containing information necessary to decode an image (parameter set or SEI message).

[0097] The aforementioned VCL NAL unit and Non-VCL NAL unit can be transmitted over a network by attaching header information according to the data specifications of the underlying system. For example, the NAL unit can be transformed into a data format of a specified specification, such as H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

[0098] As described above, the NAL unit type can be determined according to the RBSP data structure included in the NAL unit, and information about this NAL unit type can be stored in the NAL unit header and signaled.

[0099] For example, NAL units can be broadly classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether they contain information about the image (slice data). VCL NAL unit types can be classified according to the properties and types of the picture included in the VCL NAL unit, while Non-VCL NAL unit types can be classified according to the types of parameter sets.

[0100] The following is an example of a NAL unit type specified according to the type of parameter set included in the Non-VCL NAL unit type.

[0101] - APS (Adaptation Parameter Set) NAL unit: Type for the NAL unit containing the APS

[0102] - DPS(Decoding Parameter Set) NAL unit: Type for the NAL unit containing the DPS

[0103] - VPS (Video Parameter Set) NAL unit: Type for the NAL unit containing the VPS

[0104] - SPS (Sequence Parameter Set) NAL unit: Type for the NAL unit containing the SPS

[0105] - PPS(Picture Parameter Set) NAL unit: Type for the NAL unit containing the PPS

[0106] The above-described NAL unit types have syntax information for the NAL unit type, and said syntax information can be stored in the NAL unit header and signaled. For example, said syntax information may be nal_unit_type, and NAL unit types may be specified by the nal_unit_type value.

[0107] A slice header (slice header syntax, slice header information) may include information / parameters that can be commonly applied to the slice. The APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be commonly applied to one or more slices or pictures. The SPS (SPS syntax) may include information / parameters that can be commonly applied to one or more sequences. The VPS (VPS syntax) may include information / parameters that can be commonly applied to multiple layers. The DPS (DPS syntax) may include information / parameters that can be commonly applied to the entire video. The DPS may include information / parameters related to the concatenation of a CVS (coded video sequence). In the present disclosure, High Level Syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, or slice header syntax.

[0108] In the present disclosure, image / video information encoded by an encoding device and signaled in the form of a bitstream includes not only information related to picture partitioning, intra / inter prediction information, residual information, in-loop filtering information, etc., but may also include information included in the slice header, information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and / or information included in the DPS.

[0109] The following descriptor of the present disclosure specifies the parsing process for each syntax element:

[0110] - ae(v): context-adaptive arithmetic entropy-coded syntax element.

[0111] - b(8): A byte (8 bits) with an arbitrary bit sequence pattern. The parsing process for this descriptor is specified by the return value of the function read_bits(8).

[0112] - f(n): A fixed-pattern bit string using n bits written with the left bit first. The parsing process for this descriptor is specified by the return value of the function read_bits(n).

[0113] - i(n): A signed integer using n bits. In the syntax table, if n is "v", the number of bits depends on the values ​​of other syntax elements. The parsing process for this descriptor is specified by the return value of the function read_bits(n), which is interpreted as a two's complement integer representation where the most significant bit is written first.

[0114] - se(v): A signed integer 0th order Exp-Golomb-coded syntax element with the left bit coming first. The parsing process for this descriptor is specified as having order k of 0.

[0115] - st(v): A null-terminated string encoded in Universal Coded Character Set (UCS) Transfer Format-8 (UTF-8) characters as specified in ISO / IEC 10646. The parsing process is specified as follows: st(v) reads and returns a sequence of bytes from the bitstream starting at the byte-aligned position of the bitstream, from the current position to a point that does not contain the next byte-aligned byte, such as 0x00, and moves the bitstream pointer by (stringLength + 1) * 8 bit positions, where stringLength is equal to the number of bytes returned.

[0116] For reference, the st(v) syntax descriptor is used in this specification only when the current position of the bitstream is a byte alignment position.

[0117] - tu(v): A truncated unary code using up to maxVal bits, using maxVal defined in the semantics of the syntax element.

[0118] - u(n): An unsigned integer using n bits. In the syntax table, if n is "v", the number of bits depends on the values ​​of other syntax elements. The parsing process for this descriptor is specified by the return value of the function read_bits(n), which is interpreted as the binary representation of the unsigned integer with the most significant bit written first.

[0119] - ue(v): An unsigned integer 0th-order Exp-Golomb-coded syntax element with the left bit coming first. The parsing process for this descriptor is specified as having order k of 0.

[0120] The SEI message related to the present disclosure will be described below.

[0121] Table 1 shows an example of SEI message syntax for SEI processing order according to one embodiment.

[0122] [Table 1]

[0123]

[0124] An example of the semantics of an SEI message in an SEI processing sequence according to one embodiment is described.

[0125] SEI Processing Order (SPO): SEI messages convey information indicating the preferred processing order determined by the encoder (i.e., content producer) for a group of SEI messages that may exist within the Encoding Layer Video Sequence (CLVS).

[0126] The use of this SEI message requires the following definition:

[0127] - Two lists of payload type values, SeiProcessingOrderSeiList and SpoProcessSeiList.

[0128] The semantics of SPO SEI messages utilize the concept of types of SEI messages. SEI messages with different payloadType values ​​are considered to be of different types of SEI messages. Additionally, different SEI messages that have the same payloadType value but are distinguished by the syntax elements within the SEI payload are considered to be of different types of SEI messages. This distinction based on the syntax elements within the SEI payload must be performed by comparing values ​​transmitted using po_sei_prefix_data_bit[i][j] syntax elements (if present) or values ​​transmitted as SEI messages within a processing order nesting SEI message (if present). For example, Neural Network Post-Filter Feature (NNPFC) SEI messages can be distinguished by having different nnpfc_id values.

[0129] When the i-th SEI message seiA within any SPO SEI message has both po_sei_wrapping_flag[i] and po_sei_prefix_flag[i] set to 0, there must not exist another SEI message seiB within the same SPO SEI message or within another SPO SEI message in the current CLVS such that all of the following are true:

[0130] - The value of po_sei_payload_type[i] in seiB is the same as the value in seiA.

[0131] - The value of po_sei_wrapping_flag[i] of seiB is 0.

[0132] - The value of po_sei_prefix_flag[i] of seiB is 1.

[0133] When an SPO SEI message with a specific po_id value exists in any picture unit of the CLVS, that SPO SEI message with the specific po_id value must exist in the first picture unit of the CLVS in decoding order. The number of SEI messages and the payload type codes of the SEI messages indicated within each SPO SEI message with the same po_id value persist in decoding order from the current picture unit to the end of the CLVS in output order.

[0134] An SPO SEI message may carry one or more SEI prefix indications for a specific payloadType. When present, each SEI prefix indication is a bit sequence following the SEI payload syntax of the corresponding payloadType value, containing multiple complete syntax elements starting from the first syntax element of the SEI payload. These SEI prefix indications must provide sufficient information to determine a specific processing order for SEI message types that have the same payloadType value but different preferred processing orders.

[0135] po_id includes an identification number for identifying SPO SEI messages.

[0136] A processing chain consists of a list of SEI message types identified by the SPO SEI message according to the preferred processing order indicated in the SPO SEI message. When multiple processing chains are directed by SPO SEI messages for the same access unit, the decoder must operate only one of the directed processing chains.

[0137] Each SEI message type within the processing chain indicated by the SPO SEI message is identified by the syntax elements po_sei_payload_type[i], po_sei_wrapping_flag[i], po_sei_processing_order[i], and, if present, po_num_bits_in_prefix_indication_minus1[i] and po_prefix_data_bit[i][j].

[0138] SEI message types do not need to belong to any processing chain and can belong to any number of processing chains identified by SPO SEI messages having different po_id values.

[0139] Each SEI message of the SEI message type identified within the SPO SEI message has the same persistence scope as when the corresponding SEI message is transmitted outside the SPO SEI message and is not identified within the SPO SEI message.

[0140] For reference, if an SEI message specifies a process and is not associated with a processing chain specified by any SPO SEI message, it is implicitly a processing chain in itself. Some standards, such as Rec. ITU-T H.266 | ISO / IEC 23090-3, specify an implicit processing chain where another NNPF follows a super-resolution NNPF. Implicitly specified processing chains are treated like processing chains specified by SPO SEI messages when selecting the SEI messages to apply.

[0141] po_for_human_viewing_idc identical to 3 indicates that the intended optimal use of the video resulting from the processing chain specified by this SPO SEI message includes human viewing. po_for_human_viewing_idc identical to 2 indicates that the video resulting from the processing chain specified by this SPO SEI message is suitable but not specifically optimized for human viewing. po_for_human_viewing_idc identical to 1 indicates that the video resulting from the processing chain specified by this SPO SEI message is unsuitable for human viewing. po_for_human_viewing_idc identical to 0 indicates that it is impossible to know whether the video resulting from the processing chain specified by this SPO SEI message is suitable for human viewing.

[0142] po_for_machine_analysis_idc identical to 3 indicates that the intended optimal use of the video resulting from the processing chain specified by this SPO SEI message includes machine analysis. po_for_machine_analysis_idc identical to 2 indicates that the video resulting from the processing chain specified by this SPO SEI message is suitable but not specifically optimized for machine analysis. po_for_machine_analysis_idc identical to 1 indicates that the video resulting from the processing chain specified by this SPO SEI message is unsuitable for machine analysis. po_for_machine_analysis_idc identical to 0 indicates that it is impossible to know whether the video resulting from the processing chain specified by this SPO SEI message is suitable for machine analysis.

[0143] The requirement for bitstream conformance is that the values ​​of po_for_human_viewing_idc and po_for_machine_analysis_idc must not both be equal to 1.

[0144] For reference, the values ​​of po_for_human_viewing_idc and po_for_machine_analysis_idc are valid for the output of the processing chain.

[0145] po_reserved_zero_4bits must be 0. Values ​​greater than 0 for po_reserved_zero_4bits are reserved for future use and must not exist in a bitstream suitable for this version of this specification. Decoders suitable for this version of this specification must accept any value of po_reserved_zero_4bits within the range of 0 to 15 (inclusive).

[0146] po_num_sei_messages_minus2 plus 2 represents the number of SEI message types that indicate the preferred order of processing within the SPO SEI message. The variable PoNumProcStgs is set to equal po_num_sei_messages_minus2 + 2.

[0147] po_breadth_first_flag equal to 1 specifies that breadth-first handling of the processing chain should be applied to determine the pictures used to interpret the semantics of SEI messages applied as part of the processing chain specified by this SPO SEI message. po_breadth_first_flag equal to 0 specifies that breadth-first handling of the processing chain or depth-first handling of the processing chain should be applied to determine the pictures used to interpret the semantics of SEI messages applied as part of the processing chain specified by this SPO SEI message.

[0148] For reference, when po_breadth_first_flag is equal to 0, the processing chain can be executed on the picture without processing any SEI messages applied to subsequent picture units in output order.

[0149] po_sei_wrapping_flag[i], identical to 1, specifies that the SEI message applied as the i-th SEI message type within the processing chain specified in this SPO SEI message (if any) is a SEI message included in a PON SEI message where all of the following conditions are true:

[0150] - pon_target_po_id[j] with any value j is equal to po_id.

[0151] - Within the processing order nesting SEI message, there exists a k-th loop entry where the payload type of the k-th nested SEI message is the same as po_sei_payload_type[i] and pon_processing_order[k] is the same as po_sei_processing_order[i].

[0152] po_sei_wrapping_flag[i], which is equal to 0, specifies that the SEI message applied as the i-th SEI message type within the processing chain specified in this SPO SEI message (if any) is a SEI message that is not included in the PON SEI message and that all of the following conditions are true:

[0153] - The payload type of the corresponding SEI message is the same as po_sei_payload_type[i].

[0154] - When po_sei_prefix_flag[i] is equal to 0 or po_sei_prefix_flag[i] is equal to 1, the payload of the corresponding SEI message starts with the values ​​of po_sei_prefix_data_bit[i][j].

[0155] For reference, po_sei_wrapping_flag[i], which is equal to 1, enables SEI messages to be passed within processing order nesting SEI messages to prevent such SEI messages from being misinterpreted by decoders that do not process SPO SEI messages. Therefore, po_sei_wrapping_flag[i], which is equal to 1, is intended to be used in cases where unintended results may be produced by such decoders when po_sei_wrapping_flag[i] is 0.

[0156] po_sei_importance_flag[i] influences the derivation of PoSeiList, which is a list of SEI messages that the decoding system must process for a specific picture picA, as specified below.

[0157] po_sei_processing_degree_flag[i] affects the derivation of PoSeiList as specified below.

[0158] A processing chain may contain zero or more sub-chains. A sub-chain contains SEI message types such that all of these SEI message types must be processed by the decoding system, or if the decoding system cannot interpret or does not support one or more of these SEI message types, then no SEI message types in that sub-chain should be processed. The SEI message types belonging to a sub-chain are determined by the values ​​of po_importance_flag[i] and po_processing_degree_flag[i] as specified below.

[0159] Table 2 specifies the interpretation of po_importance_flag[i] and po_processing_degree_flag[i].

[0160] [Table 2]

[0161]

[0162] po_sei_payload_type[i] specifies the payload type value of the i-th type SEI message.

[0163] For reference, the NNPFC SEI message type has a po_sei_payload_type[i] value that is the same as the payload type value of the NNPFC SEI message. Examples of reasons for including the NNPFC SEI message type in the SPO SEI message include the following:

[0164] - The NNPFC SEI message type includes an SEI prefix containing the nnpfc_purpose syntax element, which provides a hint to the decoding system about what kind of processing is included in each NNPF within the processing chain.

[0165] - NNPF must be called only as part of the processing chain, so that the NNPFC SEI message type is associated with po_sei_wrapping_flag[i] which is the same as 1, and each NNPFC SEI message(s) is included in the PON SEI message(s).

[0166] When po_sei_payload_type[i] represents an NNPFC SEI message for a specific NNPF, the same SPO SEI message must contain po_sei_payload_type[j] representing each NNPFA SEI message for the same NNPF for j greater than i.

[0167] po_sei_prefix_flag[i] equal to 1 indicates that the syntax elements po_num_bits_in_prefix_indication_minus1[i] and some po_sei_prefix_data_bit[i][j] exist. po_sei_prefix_flag[i] equal to 0 indicates that these syntax elements do not exist.

[0168] For each i in the range from 0 to po_num_sei_messages_minus2 + 1 (inclusive), the value of po_sei_payload_type[i] must be equal to the value in SeiProcessingOrderSeiList.

[0169] When po_sei_payload_type[i] is equal to any value in SpoProcessSeiList, the SEI message of the i-th type represents a process.

[0170] spoPropertySeiList is configured to consist of payload type values ​​included in SeiProcessingOrderSeiList, excluding payload type values ​​included in SpoProcessSeiList. When po_sei_payload_type[i] is equal to any value in spoPropertySeiList, the SEI message of the i-th type represents a property.

[0171] po_sei_processing_order[i] represents the preferred order of processing for the i-th type of SEI message, for which information on the preferred processing order in SPO SEI messages is provided. For any two distinct integer values ​​of m and n, if po_sei_processing_order[m] is less than po_sei_processing_order[n], it indicates that the type of SEI message associated with index m must be processed before the type of SEI message associated with index n; if po_sei_processing_order[m] is equal to po_sei_processing_order[n], it indicates that there is no preferred order of processing between the types of SEI messages associated with indices m and n (for example, they may represent different attributes applicable at the corresponding stage, or one may represent an attribute and the other a process).

[0172] For i greater than 0, po_sei_processing_order[i] must be greater than or equal to po_sei_processing_order[i - 1].

[0173] seiMsgA is defined as the SEI message that is applied as the i-th SEI message type in the processing chain specified in this SPO SEI message, persists for a specific picture picA, and is associated with po_sei_processing_order[i], such as poValA.

[0174] seiMsgSet is defined as a set of SEI messages consisting of each SEI message for which all of the following conditions are true:

[0175] - The corresponding SEI message is applied as the k-th SEI message type having an arbitrary value k smaller than i in the processing chain specified in this SPO SEI message.

[0176] - The corresponding SEI message persists for picA.

[0177] - po_sei_processing_order[k] is smaller than poValA.

[0178] - The payload type of the corresponding SEI message is among the values ​​included in SpoProcessSeiList.

[0179] The pictures to which the semantics of seiMsgA apply are specified as follows:

[0180] - If seiMsgSet is not empty, the semantics of seiMsgA apply to all pictures generated by the process implied by the SEI message having the largest value of po_sei_processing_order[k] among the SEI messages in seiMsgSet.

[0181] - Otherwise, the semantics of seiMsgA apply to picA.

[0182] For reference, when an NNPF process outputs one or more NNPF-generated pictures, the semantics of the SEI message following the NNPF in the processing order apply to all of these NNPF-generated pictures.

[0183] po_num_bits_in_prefix_indication_minus1[i] and po_sei_prefix_data_bit[i][j] have the same semantics as the num_bits_in_prefix_indication_minus1[i] and sei_prefix_data_bit[i][j] syntax elements of the SEI prefix indication SEI message, with prefix_sei_payload_type replaced by po_sei_payload_type[i] if present.

[0184] When there is one or more SPO SEI messages with a specific po_id value within CLVS, the values ​​of po_for_human_viewing_idc, po_for_machine_analysis_idc, po_num_sei_messages_minus2, and po_breadth_first_flag, and the values ​​of po_sei_wrapping_flag[i], po_sei_importance_flag[i], po_sei_processing_degree_flag[i], po_sei_payload_type[i], po_sei_prefix_flag[i], and po_sei_processing_order[i] for each value i must be equal to the values ​​in other SPO SEI messages within CLVS that have the same po_id value.

[0185] po_byte_alignment_bit_equal_to_one must be equal to 1.

[0186] The list PoProcStgIdx, representing the processing step indices of SEI message types within the processing chain, and the list PoSeiTypeIdx, representing the processing step indices of SEI message types within the processing chain, are derived as follows:

[0187] - For each SEI message type in the processing chain, with j initially set to 0, the following is applied in the non-decreasing order of the corresponding po_sei_processing_order[i] values:

[0188] [Formula 1]

[0189]

[0190] Here, PoProcStgIdx[i] represents the processing stage index of the i-th SEI message type within the processing chain, and PoSeiTypeIdx[j] represents the SEI message type index of the j-th processing stage within the processing chain.

[0191] For j in the range from 0 to (inclusive) PoNumProcStgs - 1, the list poSubChainIdx[j], which specifies the sub-chain index of the j-th processing stage of the processing chain, is derived as follows:

[0192] [Formula 2]

[0193]

[0194] For a picture, the list PoSeiList representing the list of SEI messages that can be applied to the picture, the list PoSeiTypeList representing the SEI message type indices of the SEI messages that can be applied to the picture, and the variable PoNumSeiMsgs representing the number of SEI messages that can be applied to the picture are derived as follows:

[0195] - PoSeiList is initially empty, and seiListIdx and PoNumSeiMsgs are both initially set to 0.

[0196] - Unless terminated earlier as specified below, for all j values ​​in the range from 0 to (inclusive) PoNumProcStgs - 1, the following applies in increasing order of j:

[0197] - When the SEI message seiA associated with the PoSeiTypeIdx[j]-th SEI message type persists for picA, the following applies:

[0198] - If all of the following conditions are true, seiA is added to the end of PoSeiList, PoSeiTypeList[seiListIdx] is set to equal PoSeiTypeIdx[j], PoNumSeiMsgs is set to equal PoNumSeiMsgs + 1, and seiListIdx is set to equal seiListIdx + 1:

[0199] - The decoding system can interpret and support the functionality indicated by seiA.

[0200] - One of the following conditions is true:

[0201] - poSubChainIdx[j] is equal to 0.

[0202] - For all k values ​​where poSubChainIdx[k] is equal to poSubChainIdx[j] and po_sei_importance_flag[PoSeiTypeIdx[k]] is equal to 1 or po_sei_processing_degree_flag[PoSeiTypeIdx[k]] is equal to 1, the decoding system can interpret and support the functionality indicated by all SEI message types.

[0203] - Otherwise, if po_sei_importance_flag[PoSeiTypeIdx[j]] is equal to 1 and po_sei_processing_degree_flag[PoSeiTypeIdx[j]] is equal to 1, the processing chain specified by this SPO SEI message should not be performed for picA, PoSeiList is set to empty, PoNumSeiMsgs is set to 0, and the derivation of PoSeiList, PoSeiTypeList, and PoNumSeiMsgs is terminated.

[0204] po_complexity_info_present_flag equal to 1 specifies that there are one or more syntax elements indicating the complexity of invoking NNPFs within the processing chain identified by the SEI processing order SEI message. po_complexity_info_present_flag equal to 0 specifies that there are no syntax elements indicating the complexity of invoking NNPFs within the processing chain identified by the SEI processing order SEI message.

[0205] po_parameter_type_idc equal to 0 indicates that NNPFs within the processing chain identified by the SEI processing order SEI message use only integer parameters. po_parameter_type_flag equal to 1 indicates that NNPFs within the processing chain identified by the SEI processing order SEI message may use floating-point or integer parameters. po_parameter_type_idc equal to 2 indicates that NNPFs within the processing chain identified by the SEI processing order SEI message use only binary parameters. po_parameter_type_idc equal to 3 is reserved for future use by ITU-T | ISO / IEC and must not exist in a bitstream conforming to the present disclosure. Decoders suitable for the present disclosure must ignore SPO SEI messages having po_parameter_type_idc identical to 3.

[0206] po_log2_parameter_bit_length_minus3, which is equivalent to 0, 1, 2, and 3, indicates that the NNPFs within the processing chain identified by the SEI processing order SEI message do not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. When po_parameter_type_idc exists and po_log2_parameter_bit_length_minus3 does not exist, the NNPFs within the processing chain identified by the SEI processing order SEI message do not use parameters with a bit length greater than 1.

[0207] po_num_parameters_idc represents the maximum number of parameters required by NNPFs within the processing chain identified by the SEI processing order SEI message, in powers of 2048. po_num_parameters_idc equal to 0 indicates that the maximum number of parameters required by NNPFs within the processing chain identified by the SEI processing order SEI message is unknown. The value of po_num_parameters_idc must be within the range of 0 to 52 (inclusive). Po_num_parameters_idc values ​​greater than 52 are reserved for future use by ITU-T | ISO / IEC and must not exist in bitstreams conforming to this edition of this document. Decoders conforming to this edition of this document must ignore SPO SEI messages having po_num_parameters_idc greater than 52.

[0208] If the value of po_num_parameters_idc is greater than 0, the variable maxNumParameters is derived as follows:

[0209] [Formula 3]

[0210]

[0211] The requirement of bitstream conformance is that the number of parameters must be less than or equal to maxNumParameters.

[0212] A po_num_kmac_operations_idc greater than 0 indicates that the maximum number of multiply-accumulate operations per sample of NNPFs within the processing chain, identified by the SEI processing order SEI message, is less than or equal to po_num_kmac_operations_idc * 1000. A po_num_kmac_operations_idc equal to 0 indicates that the maximum number of multiply-accumulate operations of NNPFs within the processing chain, identified by the SEI processing order SEI message, is unknown. The value of po_num_kmac_operations_idc ranges from 0 to 2 32 - It must be within the range of up to (inclusive) 2.

[0213] po_total_kilobyte_size greater than 0 indicates the total size in kilobytes required to store uncompressed parameters for NNPFs within the processing chain identified by the SEI processing sequence SEI message. The total size in bits is greater than or equal to the sum of the bits used to store each parameter. po_total_kilobyte_size is the total size in bits divided by 8000 and rounded up. po_total_kilobyte_size equal to 0 indicates that the total size required to store parameters for NNPFs within the processing chain identified by the SEI processing sequence SEI message is unknown. The value of po_total_kilobyte_size ranges from 0 to 2 32 - It must be within the range of up to (inclusive) 2.

[0214] Table 3 shows an example of a processing order nesting SEI message syntax according to one embodiment.

[0215] [Table 3]

[0216]

[0217] An example of the semantics of a processing sequence nesting SEI message according to one embodiment is described.

[0218] A processing order nesting (PON) SEI message includes one or more SEI messages that must be applied only as part of the processing chain identified by the associated SEI processing order SEI message and must not be applied in a manner that contradicts the processing chain identified by the associated SEI processing order SEI message.

[0219] The use of this SEI message requires the following definition:

[0220] - Syntax structure of the container for SEI messages, sei_pon_nested_message()

[0221] SEI messages included in a PON SEI message are referred to as PON-nested SEI messages.

[0222] For reference, an encoder may contain multiple PON SEI messages within the same access unit. For example, the first PON SEI message within the access unit may contain a PON nested SEI message applicable to multiple processing chains, and one or more other PON SEI messages within the same access unit may contain PON nested SEI messages applicable to only a single processing chain.

[0223] It is a requirement of bitstream conformance that the semantics and effects of an SEI message other than a PON-nested SEI message must not depend on any PON-nested SEI message. The consequences of this constraint include the following specific constraints, where an associated SEI message is considered to be an SEI message that influences the semantics or effects of a specific SEI message:

[0224] - When there exists a neural-network post-filter characteristics SEI message with a specific value of nnpfc_id and this is a PON-nested SEI message, all associated neural-network post-filter activation (NNPFA) SEI messages with nnpfa_target_id equal to that specific nnpfc_id value must also be PON-nested SEI messages.

[0225] - When there exists a Neural Network Post-Filter Activation (NNPFA) SEI message with nnpfa_target_id of a specific value and this is not a PON-nested SEI message, the picture (if any) that has an NNPFA SEI message with the same value as the next picture in output order within the same Encoding Layer Video Sequence (CLVS) must not have an associated NNPFA SEI message that is a PON-nested SEI message.

[0226] - When there is a film grain characteristics SEI message with fg_characteristics_persistence_flag equal to 1 and this is not a PON-nested SEI message, there must not be any associated film grain characteristics SEI message that is a PON-nested SEI message within the same CLVS.

[0227] - When there exists a frame packing arrangement SEI message with fp_arrangement_persistence_flag equal to 1 and it is not a PON nested SEI message, there must not be an associated frame packing arrangement SEI message within the same CLVS that is a PON nested SEI message with fp_arrangement_cancel_flag equal to 1 or fp_arrangement_id of the same value.

[0228] - When a content color volume SEI message with ccv_persistence_flag equal to 1 exists and it is not a PON-nested SEI message, there must not be an associated frame packing arrangement SEI message that is a PON-nested SEI message within the same CLVS.

[0229] - When there is an equirectangular projection SEI message with erp_persistence_flag equal to 1 and it is not a PON-nested SEI message, there must not be any associated equirectangular projection SEI message that is a PON-nested SEI message within the same CLVS.

[0230] - When a region-wise packing SEI message with rwp_persistence_flag equal to 1 exists and is not a PON-nested SEI message, there must not be any associated region-wise packing SEI message that is a PON-nested SEI message within the same CLVS.

[0231] - When there is a sample aspect ratio SEI message with sari_persistence_flag equal to 1 and it is not a PON-nested SEI message, there must not be any associated sample aspect ratio SEI message that is a PON-nested SEI message within the same CLVS.

[0232] - When an annotated regions SEI message exists that is not a PON-nested SEI message, there must not be any associated annotated regions SEI message that is a PON-nested SEI message within the same CLVS.

[0233] - When an alpha channel information SEI message exists that is not a PON-nested SEI message, there must not be any associated alpha channel information SEI message that is a PON-nested SEI message within the same CLVS.

[0234] - When a display orientation SEI message exists that is not a PON nested SEI message, there must not be any associated display orientation SEI message that is a PON nested SEI message within the same CLVS.

[0235] - When there is a color transform indication SEI message with colour_transform_persistence_flag equal to 1 and it is not a PON nested SEI message, there must not be an associated color transform indication SEI message within the same CLVS that is a PON nested SEI message with colour_transform_cancel_flag equal to 1 or a colour_transform_id of the same value.

[0236] pon_num_po_ids_minus1 plus 1 specifies the number of SEI processing order SEI messages associated with this PON SEI message.

[0237] pon_target_po_id[i] represents the po_id of the i-th SEI processing sequence SEI message associated with this PON SEI message.

[0238] pon_num_seis_minus1 plus 1 specifies the number of PON nested SEI messages included in this PON SEI message.

[0239] pon_processing_order[i] specifies the position of the i-th PON-nested SEI message within the processing order defined by the associated SEI processing order SEI message. When i is greater than 0, pon_processing_order[i] must be greater than or equal to pon_processing_order[i - 1].

[0240] The associated SEI processing sequence SEI message for the i-th PON-nested SEI message is a SEI processing sequence SEI message having entry k for which all of the following conditions are true:

[0241] - po_sei_processing_order[k] is equal to pon_processing_order[i].

[0242] - po_sei_payload_type[k] is equal to the payload type value of the i-th PON-nested SEI message.

[0243] - When po_sei_prefix_flag[k] is equal to 1, po_sei_prefix_data_bit[k][j] for j in the range from 0 to po_num_bits_in_prefix_indication_minus1[k] (inclusive) contains the same content as the initial po_num_bits_in_prefix_indication_minus1[k] plus 1 bits of the SEI message payload of the i-th PON-nested SEI message.

[0244] The i-th PON nested SEI message can have any number of associated SEI processing sequence SEI messages in the range from 0 to pon_num_po_ids_minus1 + 1 (inclusive).

[0245] When the i-th PON-nested SEI message has an associated SEI processing sequence SEI message, the i-th PON-nested SEI message must be applied as the k-th loop entry of the associated SEI processing sequence SEI message.

[0246] The semantics of the i-th PON-nested SEI message applied as the k-th loop entry of an associated SEI processing sequence SEI message having a specific po_id value are applied without considering any of the PON-nested SEI messages that are not associated with any SEI processing sequence SEI message having that specific po_id value.

[0247] For each SEI processing sequence SEI message in which po_id is equal to pon_target_po_id[m] for any m value in the range from 0 to pon_num_po_ids_minus1 (inclusive) existing in CLVS, there must be at least one value n in the range from 0 to pon_num_seis_minus1 (inclusive) such that the SEI processing sequence SEI message becomes the associated SEI processing sequence SEI message for the nth PON nested SEI message.

[0248] The processing of the processing chain is explained.

[0249] Processing chains are alternatives to each other. That is, the decoding system may select at most one processing chain to be applied at a time.

[0250] The implicit NNPF cascading case is defined as the case where two neural-network post-filters (NNPFs) are both activated for a picture by NNPFA SEI messages that are not included in the PON SEI messages, one of the two NNPFs designated as nnpfA has the same nnpfc_purpose as 4, the other designated as nnpfB has multiple input pictures, and the NNPFC SEI messages of the two NNPFs are not included in the PON SEI messages. In this case, the two NNPFs are implicitly considered to belong to a single processing chain, and the processing chain contains only these two NNPFs.

[0251] For reference, the implicit NNPF chaining case can be applied when cropped decoded pictures have different spatial resolutions, which can occur, for example, when sps_ref_pic_resampling_enabled_flag is equal to 1 within a bitstream compliant with Rec. ITU-T H.266 | ISO / IEC 23090-3. In one example, the cropped decoded pictures have two spatial resolutions, and the first NNPF in the processing chain for the implicit NNPF chaining case is optionally enabled so that the low-resolution cropped decoded pictures are upsampled to the resolution of the high-resolution cropped decoded pictures. Consequently, the input pictures for the second NNPF in the processing chain have the same spatial resolution.

[0252] Except for implicit NNPF cascading cases, each processing chain containing multiple SEI message types is indicated by an SPO SEI message having a specific value of po_id. Except for implicit NNPF cascading cases, when a bitstream is not contained in PON SEI messages and has a specific payload type and that payload type contains one or more SEI messages existing in SpoProcessSeiList, the SEI message type for that payload type is within its own processing chain. This case is referred to as the implicit single-processing-stage case.

[0253] In the case of an implicit NNPF chain, po_breadth_first_flag is inferred to be equal to 0 or 1, PoNumProcStgs is set to 2, PoNumSeiMsgs is set to 2, and the following applies:

[0254] 1) SEI message types with indices 0 and 1 correspond to nnpfA and nnpfB, respectively.

[0255] 2) If both NNPFs have the same nnpfc_purpose as 4 and have multiple input pictures, one of them is selected to be applied first, and for the one selected to be applied first, PoProcStgIdx[i] is set to 0 and for the other, PoProcStgIdx[i] is set to 1.

[0256] 3) Otherwise, PoProcStgIdx[0] is set to 0 and PoProcStgIdx[1] is set to 1.

[0257] In the case of an implicit single processing step, po_breadth_first_flag is inferred to be equal to 0 or 1, PoNumProcStgs is set to 1, PoNumSeiMsgs is set to 1, PoProcStgIdx[0] is set to 0, and PoSeiList[0] is set to be a single SEI message.

[0258] The PoSeiList for the corresponding picture of picA or the associated inserted picture of picA is derived identically to the PoSeiList derived for picA.

[0259] The decoding system can select and apply the processing chain as follows:

[0260] 1) The bitstream is decoded, a processing chain is selected, and the following is applied:

[0261] - The list PoCdoPicList is configured to be a list of cropped decoded pictures in output order generated as a result of decoding the bitstream.

[0262] - The list PoDecPicList is configured to be a list of decoded pictures in output order generated as a result of decoding the bitstream.

[0263] 2) For each decoded picture picA in the output order, the following applies:

[0264] a. When an implicit NNPF cascading case is selected, the following applies to picA:

[0265] - PoNumSeiMsgs is initially set to 0 for picA.

[0266] When the NNPFA SEI message corresponding to PoProcStgIdx[i], which is equal to 0, enables NNPF for picA, PoSeiList[PoNumSeiMsgs++] is set to be equal to that NNPFA SEI message.

[0267] When the NNPFA SEI message corresponding to PoProcStgIdx[i], which is the same as 1, enables NNPF for picA, PoSeiList[PoNumSeiMsgs++] is set to be the same as the corresponding NNPFA SEI message.

[0268] b. If PoSeiTypeList[0] does not correspond to an FGC SEI message, the pictures in PoCdoPicList corresponding to picA are added to the list CandInputPicList[i] for all i values ​​in the range from 0 to PoSeiTypeList[0] (inclusive).

[0269] c. Otherwise, the picture in PoDecPicList corresponding to picA is added to the list CandInputPicList[i] for all i values ​​in the range from 0 to PoSeiTypeList[0] (inclusive).

[0270] d. Pictures in PoCdoPicList corresponding to picA are added to the list CandInputPicList[i] for all i values ​​in the range from PoSeiTypeList[0] + 1 to PoNumProcStgs (inclusive).

[0271] For reference, the list CandInputPicList[i] for i in the range from 1 to (inclusive) PoNumProcStgs can be updated during the next step. After step 4 below, the list CandInputPicList[PoNumProcStgs] stores the final output of the selected processing chain.

[0272] 3) If the selected processing chain is indicated by the SPO SEI message and the po_breadth_first_flag in the SPO SEI message is equal to 1, the breadth-first handling of the processing chain is called. Otherwise, the depth-first handling of the processing chain or the breadth-first handling of the processing chain is called, and regardless of whether the depth-first handling of the processing chain or the breadth-first handling of the processing chain is called, the resulting CandInputPicList[PoNumProcStgs] must be identical, which is a requirement of bitstream conformance.

[0273] 4) The list PoOutputPicList is set to be identical to CandInputPicList[PoNumProcStgs].

[0274] For reference, the pictures in PoOutputPicList are in increasing output order, and there are no two pictures in PoOutputPicList with the same output order.

[0275] For each processing stage with a processing stage index i in the range from 0 to PoNumProcStgs - 1 (inclusive), the following applies:

[0276] - For any specific pair of pictures, inputPicA and inputPicB, that are consecutive in output order within CandInputPicList[i], which is a list of candidate input pictures for the corresponding processing step, when applying a process implied by a specific SEI message of the processing step, if one or more intermediate pictures, intermediatePicSetA, between inputPicA and inputPicB in output order are added to CandInputPicList[i + 1], then the application of the process implied by that specific SEI message when a picture other than currPicA was the current picture, or the application of the process implied by another SEI message of the same processing step when any picture (including currPicA) was the current picture, shall not output any picture between inputPicA and inputPicB in output order.

[0277] For reference, the intention of the constraints expressed above is to prevent the generation of output pictures between any specific consecutive pair of input pictures more than once within the processing step.

[0278] Breadth-first handling of a processing chain is explained.

[0279] For each SEI message type with SEI message type index i of the selected processing chain, the following is applied in increasing order of the corresponding processing step index PoProcStgIdx[i] values:

[0280] - For each picture picA in output order within CandInputPicList[PoProcStgIdx[i]], if a SEI message associated with the i-th SEI message type exists in picA's PoSeiList, the following applies:

[0281] - When the i-th SEI message type does not correspond to PoSeiList[0], the following is applied for the interpretation of the SEI message:

[0282] - Interface variables for the purpose of interpreting SEI messages are derived from picA.

[0283] - The semantics of the SEI message, or the semantics of the NNPFC SEI message associated with the SEI message if the SEI message is an NNPFA SEI message, are applied to the pictures in CandInputPicList[PoProcStgIdx[i]].

[0284] - When the SEI payloadType value of the i-th SEI message type exists in SpoProcessSeiList, the process implied by the SEI message is executed, and each list CandInputPicList[PoProcStgIdx[i] + j] for j in the range from 1 to PoNumProcStgs - PoProcStgIdx[i] (inclusive) is updated by replacing the pictures with the corresponding processed pictures (if any) resulting from the process, and inserting other pictures (if any) resulting from the process into CandInputPicList[PoProcStgIdx[i] + j] so as to maintain the output order. If the SEI message is an FGC SEI message and is PoSeiList[0], while replacing the picture in CandInputPicList[PoProcStgIdx[i] + j] with the corresponding processed picture, the corresponding processed picture is cropped in the same way as generating a cropped decoded output picture from the corresponding decoded picture, and that cropped picture is used for replacement.

[0285] Depth-first handling of a processing chain is explained.

[0286] For each picture picA in CandInputPicList[0], the following is applied repeatedly in output order:

[0287] - For each SEI message with SEI message index seiIdx in picA's PoSeiList, the following applies in increasing order of the list indices for PoSeiList:

[0288] - When the i-th SEI message type does not correspond to PoSeiList[0], the following is applied for the interpretation of the SEI message:

[0289] Interface variables for the purpose of interpreting SEI messages are derived from the pictures in CandInputPicList[PoProcStgIdx[PoSeiTypeList[seiIdx]]].

[0290] - The semantics of the SEI message, or the semantics of the NNPFC SEI message associated with the SEI message if the SEI message is an NNPFA SEI message, apply to the pictures within CandInputPicList[PoProcStgIdx[PoSeiTypeList[seiIdx]]].

[0291] - When the SEI payloadType value of the SEI message exists in SpoProcessSeiList, the process implied by the SEI message is repeatedly invoked in output order for each picture in CandInputPicList[PoProcStgIdx[PoSeiTypeList[seiIdx]]] that is either the picture corresponding to picA or an associated inserted picture of picA. After each call to the process, each list CandInputPicList[PoProcStgIdx[PoSeiTypeList[seiIdx]] + j] for j in the range from 1 to PoNumProcStgs - PoProcStgIdx[PoSeiTypeList[seiIdx]] (inclusive) is updated by replacing the pictures with the corresponding processed pictures (if any) resulting from the process, and inserting other pictures (if any) resulting from the process into CandInputPicList[PoProcStgIdx[PoSeiTypeList[seiIdx]] + j] so as to maintain the output order. If the SEI message is an FGC SEI message and is PoSeiList[0], while replacing the picture in CandInputPicList[PoSeiTypeList[seiIdx]] + j] with the corresponding processed picture, the corresponding processed picture is cropped in the same way as generating the decoded output picture by first cropping it from the corresponding decoded picture, and that cropped picture is used for replacement.

[0292] Table 4 shows an example of a quality metric SEI message syntax according to one embodiment.

[0293] [Table 4]

[0294]

[0295] An example of the semantics of a quality indicator SEI message according to one embodiment is described.

[0296] Quality metrics SEI messages signal quality metric values ​​representing any one of the following:

[0297] - Single picture quality.

[0298] - Average quality of all pictures corresponding to CLVS.

[0299] - Quality gain of a single picture. This is the difference in quality of a single picture relative to the quality of a gain reference picture.

[0300] - Average quality gain of all pictures corresponding to CLVS.

[0301] The use of this SEI message requires the definition of the following variables:

[0302] - Chroma format indicator denoted as ChromaFormatIdc.

[0303] - Number of pictures NumPics.

[0304] - A list of picture widths and heights in units of luma samples. Here, denoted as PicWidth[i] and PicHeight[i] respectively, where i is in the range from 0 to NumPics - 1 (inclusive).

[0305] - A list of pictures TestPicList[i]. Here, i is in the range from 0 to NumPics - 1 (inclusive).

[0306] - When any qm_gain_flag[i] is equal to 1, GainRefPicList[i] is a list of gain-referenced pictures for i in the range from 0 to NumPics - 1 (inclusive).

[0307] The variables SubWidthC and SubHeightC are derived from ChromaFormatIdc.

[0308] TestPicList[i][cIdx] and ReferencePicList[i][cIdx] represent the cIdx-th sample array of the i-th picture in TestPicList and ReferencePicList, respectively.

[0309] TestPicList[i][cIdx][x][y] and ReferencePicList[i][cIdx][x][y] represent the sample at position (x, y) within the cIdx-th sample array of the i-th picture in TestPicList and ReferencePicList, respectively. Here, x ranges from 0 to ( ( cIdx = 0 ) It is in the range from PicWidth[i] : PicWidth[i] / SubWidthC ) - 1 (inclusive), and y is from 0 ( ( cIdx = = 0 ) PicHeight[i] : PicHeight[i] / SubHeightC ) - is in the range of 1 (inclusive).

[0310] currPicIdx is set to a value where the output time of TestPicList[currPicIdx] is equal to the output time of the current picture.

[0311] qm_metric_definitions_present_flag, which is equivalent to 1, indicates that information defining quality metrics exists. qm_metric_definitions_present_flag, which is equivalent to 0, indicates that information defining quality metrics does not exist.

[0312] When this SEI message is the first quality metric SEI message in CLVS in decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0313] Otherwise (if this SEI message is not the first quality indicator SEI message within the CLVS in the decoding order), it is a requirement of bitstream conformance that at least one of the following two conditions must be satisfied:

[0314] - qm_metric_definitions_present_flag is equal to 0.

[0315] - The values ​​of the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i], and qm_metric_description[i] (if present) must be equal to the respective syntax elements of the first quality metric SEI message in CLVS.

[0316] A qm_clvs_values_present_flag equal to 1 indicates that qm_clvs_metric_value[i][c] syntax elements exist. A qm_clvs_values_present_flag equal to 0 indicates that qm_clvs_metric_value[i][c] syntax elements do not exist.

[0317] A qm_pic_values_present_flag equal to 1 indicates that qm_pic_metric_value[i][c] syntax elements exist. A qm_pic_values_present_flag equal to 0 indicates that qm_pic_metric_value[i][c] syntax elements do not exist.

[0318] qm_num_metrics_minus1 plus 1 specifies the number of signaling quality metric items.

[0319] A qm_gain_enabled_flag equal to 1 indicates that the qm_gain_flag[i] syntax element exists. A qm_gain_enabled_flag equal to 0 indicates that the qm_gain_flag[i] syntax element does not exist.

[0320] qm_gain_flag[i] and qm_gain_reference_flag[i], if present, represent the interpretation of the values ​​of the syntax elements qm_clvs_metric_value[i][c] and qm_pic_metric_value[i][c]. When qm_gain_flag[i] and qm_gain_reference_flag[i] do not exist, they are inferred to be equal to 0.

[0321] qm_metric_type[i] specifies the quality metric type of the i-th item as specified in Table 5. The value of qm_metric_type[i] must be in the range of 0..8 (inclusive) or 128..255 (inclusive) within a bitstream suitable for the present disclosure. Values ​​in the range of 9..127 (inclusive) for qm_metric_type[i] are reserved for future use and must not exist within a bitstream suitable for the present disclosure.

[0322] When the value of qm_metric_type[i] is within the range of 9 .. 127, decoders suitable for the present disclosure must ignore all syntax elements for the i-th item in this syntax structure.

[0323] When the value of qm_metric_type[i] is within the range of 128 .. 255, the quality metric type is unspecified or specified by other means not specified in this disclosure.

[0324] [Table 5]

[0325]

[0326] qm_three_component_flag[i] equal to 1 indicates that there will be three component values ​​for the i-th indicator. qm_three_component_flag[i] equal to 0 indicates that there will be a single value for the i-th indicator. It is a requirement of bitstream conformance that qm_three_component_flag[i] must be equal to 0 when ChromaFormatIdc is equal to 0.

[0327] qm_metric_increasing_flag[i] equal to 1 means that a higher value of the i-th metric indicates an improvement in quality. qm_metric_increasing_flag[i] equal to 0 means that a lower value of the i-th metric indicates an improvement in quality. When not present, the value of qm_metric_increasing_flag[i] is inferred to be equal to IncreasingFlag[qm_metric_type[i]] in Table 5.

[0328] qm_full_reference_flag[i], which is equal to 1, indicates that the quality metric is a full reference quality metric calculated by comparing the pictures in TestPicList with their respective quality reference pictures. qm_full_reference_flag[i], which is equal to 0, indicates that the quality metric may or may not include comparing the pictures in TestPicList with their respective quality reference pictures. When not present, the value of qm_full_reference_flag[i] is inferred to be equal to FullReferenceFlag[qm_metric_type[i]] in Table 5.

[0329] qm_value_len_minus1_in_bytes[i] plus 1 specifies the length in bytes of the qm_pic_metric_value[i][c] syntax element. If not present, qm_value_len_minus1_in_bytes[i] is inferred to be equal to NumBytes[qm_metric_type[i]] - 1.

[0330] qm_metric_description_present_flag[i] equal to 1 indicates that qm_metric_description[i] exists. qm_metric_description_present_flag[i] equal to 0 indicates that qm_metric_description[i] does not exist.

[0331] qm_bit_equal_to_zero must be equal to 0.

[0332] qm_metric_description[i] specifies the text description for the i-th quality metric. The length of the syntax element must be less than or equal to 4097 bytes, excluding the null-terminated byte.

[0333] qm_clvs_metric_value[i][c] specifies the mean value of the i-th quality metric for the c-th component of CLVS. The length of the syntax element is 8 * (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0334] qm_pic_metric_value[i][c] specifies the value of the i-th quality metric for the c-th component of the current picture. The length of the syntax element is 8 * (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0335] The meaning of the quality metric value is determined by the values ​​of qm_metric_type[i], qm_gain_flag[i], and qm_gain_reference_flag[i].

[0336] When qm_pic_values_present_flag is equal to 1, qm_pic_metric_value[i][c] represents the picture metric value picMetricValue[i][c] of type qm_metric_type[i] described in Table 5, and the following applies:

[0337] - If qm_gain_flag[i] is equal to 0, qm_pic_metric_value[i][c] has a value derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0338] - Otherwise (qm_gain_flag[i] is equal to 1), the following applies:

[0339] - picMetricValueTest[i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0340] - picMetricValueGainRef[i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0341] - qm_pic_metric_value[i][c] has the value of picMetricValueTest[i][c] - picMetricValueGainRef[i][c].

[0342] When qm_clvs_values_present_flag is equal to 1, qm_clvs_metric_value[i][c] represents listPicMetricValue[j][i][c], the mean value of picture metric values ​​calculated for all pictures in TestPicList of type qm_metric_type[i] as described in Table 5. Here, each listPicMetricValue[j][i][c] is derived as follows for each value of j in the range from 0 to NumPics - 1 (inclusive):

[0343] - If qm_gain_flag[i] is equal to 0, listPicMetricValue[j][i][c] is equal to picMetrictValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[j], PicWidth[j], and PicHeight[j], respectively.

[0344] - Otherwise (qm_gain_flag[i] is equal to 1), the following applies:

[0345] - listPicMetricValueTest[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0346] - picMetricValueGainRef[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0347] - qm_pic_metric_value[j][i][c] has the value of picMetricValueTest[j][i][c] - picMetricValueGainRef[j][i][c].

[0348] The derivation process of picMetricValue[i][c] is explained.

[0349] The inputs for this process are the tested picture testPic, the picture width picWidth in luma samples, and the picture height picHeight in luma samples.

[0350] The quality reference picture, referencePic, is given as input to the encoding system and is called the original picture, having an output time equal to the output time of testPic.

[0351] testPic[cIdx] and referencePic[cIdx] represent the cIdx-th sample arrays of testPic and referencePic, respectively.

[0352] testPic[cIdx][x][y] and referencePic[cIdx][x][y] represent the sample at position (x, y) within the cIdx-th sample array of testPic and referencePic, respectively.

[0353] The picture quality metric picMetricValue[i][c] is derived as follows:

[0354] - When qm_metric_type[i] is equal to 0,

[0355] - listPicMetricValueTest[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0356] - picMetricValueGainRef[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0357] - When qm_metric_increasing_flag[i] is equal to 1, a higher value of picMetricValue[i][c] indicates that testPic is of better quality than a picture with a lower picMetricValue[i][c] value.

[0358] - When qm_full_reference_flag[i] is equal to 1, picMetricValue[i][c] represents the quality metric value calculated from the comparison between testPic and referencePic.

[0359] - qm_metric_description[i] provides a text description of the quality metric displayed by picMetricValue[i][c].

[0360] - Further interpretation of picMetricValue[i][c] is determined by external means not specified in this specification.

[0361] - When qm_metric_type[i] is equal to 1,

[0362] - picMetricValue[i][0] is set to the same as the PSNR value calculated using Section 9.4.2 and Section D.2 of ISO / IEC 23001-11 [2] for the lumina components of testPic and referencePic with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0363] - When qm_three_component_flag[i] is equal to 1, - picMetricValue[i][1] and picMetricValue[i][2] are set to have the bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are equal to the PSNR values ​​calculated for the Cb and Cr components of testPic and referencePic, respectively, using sections 9.4.2 and D.2 of ISO / IEC 23001-11 [2].

[0364] - When qm_metric_type[i] is equal to 2,

[0365] - picMetricValue[i][0] is set to the value of the variable psnrYUV calculated as follows:

[0366] - The variable psnrY is set to the same as the PSNR value calculated using Section 9.4.2 and Section D.2 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0367] - Variables psnrU and psnrV are set to have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are equal to the PSNR values ​​calculated using sections 9.4.2 and D.2 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0368] [Equation 4]

[0369]

[0370] - When qm_metric_type[i] is equal to 3,

[0371] - picMetricValue[i][0] is set to the same as the SSIM value calculated from Section 9.4.2 and Section D.5 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic, with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0372] - When qm_three_component_flag[i] is equal to 1,

[0373] - picMetricValue[i][1] and picMetricValue[i][2] have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are set to be equal to the SSIM values ​​calculated from Section 9.4.2 and Section D.5 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0374] - When qm_metric_type[i] is equal to 4,

[0375] - picMetricValue[i][0] is set to the same as the MS-SSIM value calculated from Section 4.3.3 of ISO / IEC 23001-10 [1] for the luminance components of testPic and referencePic with the bit depth OrigBitDepth.

[0376] - When qm_three_component_flag[i] is equal to 1,

[0377] - picMetricValue[i][1] and picMetricValue[i][2] have a bit depth of OrigBitDepth and are set to be equal to the MS-SSIM values ​​calculated from Section 4.3.3 of ISO / IEC 23001-10 [1] for the Cb and Cr components of testPic and referencePic, respectively.

[0378] - When qm_metric_type[i] is equal to 5,

[0379] - picMetricValue[i][0] is set to the same as the MOS value specified in Section 4.3.6 of ISO / IEC 23001-10 [1].

[0380] - When qm_metric_type[i] is equal to 6,

[0381] - picMetricValue[i][0] is set to the same as the wPSNR value calculated from Section 9.4.2 and Section D.3 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic, with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0382] - When qm_three_component_flag[i] is equal to 1,

[0383] - picMetricValue[i][1] and picMetricValue[i][2] have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are set to be equal to the wPSNR values ​​calculated from Section 9.4.2 and Section D.3 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0384] - When qm_metric_type[i] is equal to 7,

[0385] - picMetricValue[i][0] is set to the same WS-PSNR value calculated from Section 9.4.2 and Section D.4 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0386] - When qm_three_component_flag[i] is equal to 1,

[0387] - picMetricValue[i][1] and picMetricValue[i][2] have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are set to be equal to the WS-PSNR values ​​calculated from Section 9.4.2 and Section D.4 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0388] - When qm_metric_type[i] is equal to 8,

[0389] - picMetricValue[i][0] is set to the value of lumaMse derived as follows, which is interpreted as a floating-point value:

[0390] [Formula 5]

[0391]

[0392] - When qm_three_component_flag[i] is equal to 1, picMetricValue[i][1] and picMetricValue[i][2] are set to the values ​​of CbMse and CrMse, respectively, derived as follows, which are interpreted as floating-point values:

[0393] [Equation 6]

[0394]

[0395] - When qm_metric_type[i] is equal to 9,

[0396] - picMetricValue[i][0] is set to equal the VMAF calculated from Section 4.3.3 of ISO / IEC 23001-10 [1] for the luminance components of testPic and referencePic with the bit depth OrigBitDepth. VMAF is a full reference metric that uses machine learning to fuse scores from multiple elementary quality metrics to generate a quality score for the video. This metric is trained to simulate a quality evaluation obtained as a result of a subjective test. The range of values ​​obtained from the standard (HD) VMAF model is [0,..100].

[0397] - When qm_metric_description_present_flag[i] is equal to 1, qm_metric_description[i] may contain information about the version of the VMAF metric, such as the metric release number and the VMAF model.

[0398] The use of the quality metric SEI message is explained.

[0399] For the purpose of interpreting quality metric SEI messages, the derivation of the variables ChromaFormatIdc, NumPics, TestPicList, PicWidth, PicHeight, and GainRefPicList is specified as follows.

[0400] TestPicList is said to initially consist of cropped decoded pictures of the current CLVS in output order.

[0401] When a quality indicator SEI message is included as an SEI message type within an SEI processing order SEI message, any quality indicator SEI message associated with that SEI processing order SEI message must be included within a processing order nesting SEI message.

[0402] When a quality metric SEI message is included as the i-th SEI message type within an SEI processing sequence SEI message, the following applies:

[0403] - It is a bitstream conformance requirement that the SEI message seiB, which implies post-processing to be performed, must exist as the j-th SEI message type within the same SEI processing order SEI message, and that po_sei_processing_order[j] must be equal to po_sei_processing_order[i].

[0404] - The corresponding quality indicator SEI message represents the picture quality resulting from the post-processing implied by seiB.

[0405] - TestPicList is updated as follows for each post-processing step where po_sei_processing_order[j] is less than or equal to po_sei_processing_order[i], in the non-decreasing order of j.

[0406] - When the post-processing step generates a picture picA as a result that has the same output time as any picture picB in TestPicList, picB in TestPicList is replaced with picA.

[0407] - When the post-processing step generates a picture picA as a result that has an output time not equal to the output time of any picture in TestPicList, picA is inserted into TestPicList in a manner that maintains the output order of the pictures in TestPicList.

[0408] NumPics is set to be equal to the number of pictures in TestPicList.

[0409] PicWidth[i] and PicHeight[i] are set to be equal to the width and height of TestPicList[i], respectively, in units of luma samples.

[0410] When a quality metric SEI message is included as the k-th SEI message within a processing order nested SEI message, and qm_gain_flag[i] is equal to 1 for any value of i, the following applies:

[0411] - If qm_gain_reference_flag[i] is equal to 0, the i-th metric value in the quality metric SEI message represents the gain of the post-processing step where po_sei_processing_order[j] is equal to pon_processing_order[k] for the picture or pictures used as input to the corresponding post-processing step, and GainRefPicList is set to be equal to TestPicList derived for the processing steps up to that point, without including the processing step with po_sei_processing_order[j].

[0412] - Otherwise (qm_gain_reference_flag[i] is equal to 1), the i-th metric value in the quality metric SEI message represents the cumulative gain of all post-processing steps for the clipped decoded picture or pictures for which po_sei_processing_order[j] is less than or equal to pon_processing_order[k], and GainRefPicList consists of the clipped decoded pictures of the current CLVS in output order.

[0413] - Bitstream conformance requirements state that the number of pictures in GainRefPicList must be equal to NumPics, and that the width and height of GainRefPicList[i] in luma samples must be equal to PicWidth[i] and PicHeight[i], respectively.

[0414] The value of ChromaFormatIdc is derived as follows:

[0415] - If the quality metric SEI message is not included within the processing order nesting SEI message, ChromaFormatIdc is set to equal sps_chroma_format_idc.

[0416] - Otherwise (when the quality metric SEI message is included within the processing order nesting SEI message), ChromaFormatIdc is set to a value that matches the chroma format of the pictures in TestPicList, and it is a bitstream conformance requirement that the chroma formats of all pictures in TestPicList and GainRefPicList (if present) must be identical.

[0417] When the quality metric SEI message qmSeiA exists in a picture unit other than the first picture unit of CLVS in decoding order and at least one value of qm_clvs_metric_value[i][c] exists, the following applies:

[0418] - If qmSeiA is not included in the processing order nesting SEI message, each value of qm_clvs_metric_value[i][c] must be equal to the value of qm_clvs_metric_value[i][c] in the quality metric SEI message that exists in the first picture unit of CLVS and is not included in the processing order nesting SEI message.

[0419] - Otherwise (qmSeiA is included within a processing order nesting SEI message containing a specific set of pon_target_po_id[i] values), the following applies:

[0420] - It is a requirement of bitstream conformance that there must be a quality indicator SEI message qmSeiB contained within a processing order nesting SEI message that exists in the first picture unit of CLVS and has the same set of pon_target_po_id[i] values.

[0421] - Each value of qm_clvs_metric_value[i][c] in qmSeiA must be equal to the value of qm_clvs_metric_value[i][c] in qmSeiB.

[0422] Conventional Quality Metric (QM) SEI messages allow QM SEI messages to be included in SEI Processing Order (SPO) SEI messages. In such cases, the following text applies:

[0423] When a quality metric SEI message is included as an SEI message type within an SEI processing order SEI message, all quality metric SEI messages associated with that SEI processing order SEI message must be included in a Processing Order Nesting (PON) SEI message.

[0424] ...

[0425] If this SEI message is the first quality metric SEI message in the layered video sequence (CLVS) coded in decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0426] Otherwise (if this SEI message is not the first quality indicator SEI message within the CLVS in the decoding order), the bitstream conformance requirement is that at least one of the following two conditions must be satisfied:

[0427] - qm_metric_definitions_present_flag is equal to 0.

[0428] - If the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i] and qm_metric_description[i] exist, their values ​​must be equal to each syntax element in the first quality metric SEI message within CLVS.

[0429] ...

[0430] If the quality metric SEI message qmSeiA exists in a picture unit other than the first picture unit of CLVS in decoding order and at least one value of qm_clvs_metric_value[i][c] exists, the following applies:

[0431] - If qmSeiA is not included in the processing order nesting (PON) SEI message, the value of each qm_clvs_metric_value[i][c] must be equal to the value of qm_clvs_metric_value[i][c] of the quality metric SEI message that exists in the first picture unit of CLVS and is not included in the processing order nesting SEI message.

[0432] - Otherwise (if qmSeiA is included in a processing order nesting SEI message with a specific set of pon_target_po_id[i] values), the following applies:

[0433] - A requirement for bitstream conformance is that there must be a quality indicator SEI message qmSeiB contained in a processing order nesting SEI message that exists in the first picture unit of CLVS and has the same set of pon_target_po_id[i] values.

[0434] - Each qm_clvs_metric_value[i][c] value of qmSeiA must be equal to the qm_clvs_metric_value[i][c] value of qmSeiB.

[0435] ...

[0436] Conventional Quality Management (QM) SEI messages are unclear and open to different interpretations:

[0437] - The first part of the above constraints covers only cases where the QM SEI message is a non-PON-nested SEI message. Essentially, this establishes that all non-PON-nested QM SEI messages are part of the same QM SEI message, and that only the values ​​of the quality indicators may differ, while the descriptions cannot. However, there is no mention of PON-nested QM SEI messages.

[0438] Even if we assume that PON-nested QM SEI messages have a different scope than non-PON-nested QM SEI messages, it remains unclear whether all PON-nested QM SEI messages have the same scope or if the scope is determined by membership in the processing chain.

[0439] In one embodiment, the following items may be applied individually or in combination.

[0440] It is specified that PON-unnested QM SEI messages have different ranges / domains than PON-nested QM SEI messages.

[0441] In the case of PON-nested QM SEI messages, all messages are part of the same domain / range.

[0442] Alternatively, for PON-nested QM SEI messages, a PON-nested QM SEI message that is part of the same processing chain (i.e., included in a specific SEI processing sequence SEI message) has a different scope / domain than a PON-nested QM SEI message that is part of a different processing chain.

[0443] An example of the semantics of a quality indicator SEI message according to one embodiment is described. Semantics different from the semantics of the quality indicator SEI message described above are described, and the description of semantics identical to the semantics of the quality indicator SEI message described above is replaced by the description of the semantics of the quality indicator SEI message described above.

[0444] ...

[0445] Quality indicator SEI messages can exist as PON-nested SEI messages and PON-unnested SEI messages.

[0446] For non-PON-nested QM SEI messages, the following applies:

[0447] - If this SEI message is the first PON-unnested quality metric SEI message in CLVS in decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0448] - Otherwise (if this SEI message is not the first PON-nested quality indicator SEI message within the CLVS in the decoding order), the bitstream conformance requirement is that at least one of the following two conditions must be satisfied:

[0449] - qm_metric_definitions_present_flag is equal to 0.

[0450] - If the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i] and qm_metric_description[i] exist, their values ​​must be equal to the respective syntax elements in the first PON-unnested quality metric SEI message within the CLVS.

[0451] For PON-nested QM SEI messages, the following applies:

[0452] - If this SEI message is the first PON-nested quality metric SEI message in CLVS in decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0453] - Otherwise (if this SEI message is not the first PON-nested quality indicator SEI message within the CLVS in the decoding order), the requirement for bitstream conformance is that at least one of the following two conditions must be satisfied:

[0454] - qm_metric_definitions_present_flag is equal to 0.

[0455] - If the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i] and qm_metric_description[i] exist, their values ​​must be equal to the respective syntax elements in the first PON-nested quality metric SEI message within the CLVS.

[0456] ...

[0457] An example of the semantics of a quality indicator SEI message according to one embodiment is described. Semantics different from the semantics of the quality indicator SEI message described above are described, and the description of semantics identical to the semantics of the quality indicator SEI message described above is replaced by the description of the semantics of the quality indicator SEI message described above.

[0458] ...

[0459] Quality indicator SEI messages can exist as PON-nested SEI messages and PON-unnested SEI messages.

[0460] For non-PON-nested QM SEI messages, the following applies:

[0461] - If this SEI message is the first PON-unnested quality metric SEI message in CLVS in decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0462] - Otherwise (if this SEI message is not the first PON-nested quality indicator SEI message within the CLVS in the decoding order), the bitstream conformance requirement is that at least one of the following two conditions must be satisfied:

[0463] - qm_metric_definitions_present_flag is equal to 0.

[0464] - If the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i] and qm_metric_description[i] exist, their values ​​must be equal to the respective syntax elements in the first PON-unnested quality metric SEI message within the CLVS.

[0465] For a PON-nested QM SEI message, where ponQMSeiA is a PON-nested QM SEI message that exists in the PON SEI message ponA associated with the SEI processing sequence SEI message spoA and is included in the processing chain directed by spoA, the following applies:

[0466] - If this SEI message is the first PON-nested quality metric SEI message within CLVS that is part of the processing chain directed by spoA in the decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0467] - Otherwise (if this SEI message is not the first PON-nested quality indicator SEI message within the CLVS that is part of the processing chain directed by spoA in the decoding order), the requirement for bitstream conformance is that at least one of the following two conditions must be satisfied:

[0468] - qm_metric_definitions_present_flag is equal to 0.

[0469] - If the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i] and qm_metric_description[i] exist, their values ​​must be equal to the respective syntax elements in the first PON-nested quality metric SEI message within the CLVS that is part of the processing chain directed by spoA.

[0470] ...

[0471] FIG. 5 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0472] The decoding method (S500) may include the operations described below.

[0473] The terms or names described below (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms or names described below. For example, the image information described below may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0474] The operations described below do not constitute an essential component of the decoding method according to one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of the decoding method according to one embodiment, and the previously described operations may be added.

[0475] The sequence of actions illustrated in the drawings regarding the actions to be described below is merely an example, and the actions to be described below may be performed in a different order as long as it does not contradict the causal relationship of the actions to be described.

[0476] The operations described below form a single embodiment integrally with the configurations and / or operations described above, unless they conflict with the configurations and / or operations described above, and do not form a separate embodiment distinct from the configurations and / or operations described above.

[0477] The terms "first," "second," "third," etc., used below are merely distinguishing indicators for identifying specific messages or information among multiple pieces of information or messages that may be included within the CLVS, and are not intended to limit the order, importance, or relative priority of these messages or information, or to restrict them to specific embodiments. For example, "first message (or first information)" refers only to at least one message (or information) included in at least one picture unit within the CLVS, and does not imply that said message (or information) must be encoded or decoded first.

[0478] The decoding method (S500) can be executed by a decoding device including a memory and a processor electrically connected to the memory, for example, by a processor.

[0479] The decoding device can acquire image information (S510).

[0480] For example, a processor of a decoding device may acquire image information. The image information may include SEI (supplemental enhancement information) messages. SEI messages may convey specific types of information that assist in processes related to the decoding, display, or other purposes of the image information. Here, SEI messages may not be necessary for the decoding process to determine the sample values ​​of the decoded picture.

[0481] For example, SEI messages may include SEI processing order (SPO) SEI messages, processing order nesting (PON) SEI messages, and / or Quality Metrics (QM) SEI messages.

[0482] The SPO SEI message may include information regarding the processing order of at least one other SEI message included in the image information (e.g., at least one SEI message other than the SPO SEI message and a PON nested SEI message included in the PON SEI message).

[0483] At least one SEI message other than the SPO SEI message and the PON nested SEI message included in the PON SEI message may have various payload types. The at least one SEI message and the PON nested SEI message may include SEI messages having different payload types and / or SEI messages having the same payload type.

[0484] The SPO SEI message may have various names, such as SEI processing order SEI message, SEI processing order related message, SEI processing order related information, processing order SEI message, processing order message, processing order related message, processing order related information, SPO message, SPO related message, and SPO related information, and such names are not limited.

[0485] SPO SEI messages can take various forms. For example, an SPO SEI message may be a syntax element or a syntax structure containing one or more syntax elements. Additionally, an SPO SEI message may be a raw byte sequence payload (RBSP) containing one or more syntax elements or one or more syntax structures. For example, an SPO SEI message may be represented as sei_processing_order( payloadSize ), but is not limited thereto.

[0486] The SPO SEI message may include identifier information of the SPO SEI message, payload type information of at least one SEI message, prefix information of at least one SEI message, and processing order information of at least one SEI message.

[0487] Identifier information may include an identifier for identifying SPO SEI messages. Image information may include multiple SPO SEI messages. Each of the multiple SPO SEI messages may include an identifier for identifying SPO SEI messages.

[0488] Identifier information may take various forms and may be represented by various names. For example, identifier information may be a syntax element or a syntax structure containing one or more syntax elements. For example, identifier information that is a syntax element may include an unsigned integer consisting of 8 bits or variable-length bits. Identifier information that is a syntax element may be represented as po_id, etc., but is not limited thereto.

[0489] Payload type information can indicate the payload type of at least one SEI message or at least one PON-nested SEI message. In other words, payload type information can specify the payloadType value of the SEI message.

[0490] Payload type information can take various forms and can be represented by various names. For example, payload type information can be a syntax element or a syntax structure containing one or more syntax elements. For example, payload type information that is a syntax element can be a 12-bit integer. Payload type information that is a syntax element can be represented as po_sei_payload_type[i], but is not limited thereto.

[0491] Prefix information may indicate whether an SPO SEI message contains prefix data for at least one SEI message. For example, prefix information with a value of 1 may indicate that an SPO SEI message contains prefix data for at least one SEI message. Additionally, prefix information with a value of 0 may indicate that an SPO SEI message does not contain prefix data for at least one SEI message. However, this is not limited thereto, and what prefix information with a value of 1 indicates may be interchangeable with what prefix information with a value of 0 indicates.

[0492] Prefix information can take various forms and can be represented by various names. For example, prefix information may be a syntax element or a syntax structure containing one or more syntax elements. For example, prefix information that is a syntax element may be a one-bit flag or an indicator with two or more bits. Prefix information that is a syntax element may be represented as po_sei_prefix_flag[i] or po_sei_prefix_idc[i], but is not limited thereto.

[0493] Processing order information may indicate the processing order for at least one payload type of SEI message. The smaller the value of the processing order information for a specific payload type, the earlier the SEI message of that payload type may be processed. In other words, if the value of the processing order information for a first payload type is smaller than the value of the processing order information for a second payload type, the SEI message of the first payload type may be processed before the SEI message of the second payload type.

[0494] Processing order information may take various forms and may be represented by various names. For example, processing order information may be a syntax element or a syntax structure containing one or more syntax elements. For example, processing order information that is a syntax element may include an unsigned integer consisting of 8 bits or variable-length bits. Processing order information that is a syntax element may be represented as po_sei_processing_order[i], etc., but is not limited thereto.

[0495] The PON SEI message may include at least one PON nested SEI message in which information regarding the processing order is provided by the SPO SEI message.

[0496] The PON SEI message may have various names, such as Processing order nesting SEI message, Processing order nesting related message, Processing order nesting related information, Processing order nesting message, Processing order nesting information, PON SEI message, PON related message, PON related information, PON message, PON information, Nesting SEI message, Nesting related message, Nesting related information, Nesting message, Nesting information, etc., and such names are not limited.

[0497] A PON SEI message can take various forms. For example, a PON SEI message may be a syntax element or a syntax structure containing one or more syntax elements. Additionally, a PON SEI message may be a raw byte sequence payload (RBSP) containing one or more syntax elements or one or more syntax structures. For example, a PON SEI message may be represented as processing_order_nesting( payloadSize ), but is not limited thereto.

[0498] A PON nested SEI message may include target identifier information, PON processing sequence information, and a PON nested SEI message.

[0499] Target identifier information may include the identifier of an SPO SEI message associated with a PON-nested SEI message. A PON-nested SEI message may be associated with multiple SPO SEI messages, and the target identifier information may include multiple target identifiers.

[0500] Target identifier information may take various forms and may be represented by various names. For example, target identifier information may be a syntax element or a syntax structure containing one or more syntax elements. For example, target identifier information that is a syntax element may include an unsigned integer consisting of 8 bits or variable-length bits. Target identifier information that is a syntax element may be represented as pon_target_po_id[i], etc., but is not limited thereto.

[0501] PON processing order information can indicate the processing order for payload types of PON-nested SEI messages. For example, PON processing order information can indicate the position of PON-nested SEI messages within the processing order defined by the associated SPO SEI message.

[0502] PON processing order information may take various forms and may be represented by various names. For example, PON processing order information may be a syntax element or a syntax structure containing one or more syntax elements. For example, PON processing order information that is a syntax element may include an unsigned integer composed of 8 bits or variable-length bits. PON processing order information that is a syntax element may be represented as pon_processing_order[i], etc., but is not limited thereto.

[0503] PON nested SEI messages are SEI messages contained within PON SEI messages, and like general SEI messages, they can convey specific types of information that assist in processes related to the decoding, display, or other purposes of image information. If a specific SEI message is not defined in some of the multiple decoding devices, that specific SEI message may be included in the PON SEI message so that undefined SEI messages are ignored by the decoding devices.

[0504] The QM SEI message may include at least one of information regarding the quality of a single picture, information regarding the average quality of all pictures corresponding to the CLVS, information regarding the quality gain of a single picture, and / or information regarding the average quality gain of all pictures corresponding to the CLVS. The information regarding quality may be expressed in various ways, such as quality information, quality indicator information, or quality indicators, and may include subjective or objective indicators regarding the quality of the picture.

[0505] QM SEI messages may have various names, such as quality indicator SEI messages, quality indicator related messages, quality indicator related information, quality indicator messages, QM messages, QM related messages, and QM related information, and such names are not limited.

[0506] QM SEI messages can take various forms. For example, a QM SEI message may be a syntax element or a syntax structure containing one or more syntax elements. Additionally, a QM SEI message may be a raw byte sequence payload (RBSP) containing one or more syntax elements or one or more syntax structures. For example, a QM SEI message may be represented as quality_metric(payloadSize), but is not limited thereto.

[0507] QM SEI messages may include picture quality existence information, average quality existence information, quality indicator definition information, quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, overall quality reference information, quality indicator length information, quality indicator description existence information, quality indicator description information, picture quality indicator information and / or average quality indicator information.

[0508] Picture quality presence information may indicate whether the QM SEI message contains picture quality indicator information for a single picture. For example, picture quality presence information with a value of 1 may indicate that the QM SEI message contains picture quality indicator information for a single picture. Additionally, picture quality presence information with a value of 0 may indicate that the QM SEI message does not contain picture quality indicator information for a single picture. However, this is not limited thereto, and what picture quality presence information with a value of 1 indicates may be different from what picture quality presence information with a value of 0 indicates.

[0509] Picture quality present information may take various forms and may be represented by various names. For example, picture quality present information may be a syntax element or a syntax structure containing one or more syntax elements. For example, picture quality present information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, picture quality present information may include qm_pic_values_present_flag or qm_pic_values_present_idc, but is not limited thereto.

[0510] Average quality presence information may indicate whether the QM SEI message contains average quality indicator information for the pictures included in CLVS. For example, average quality presence information with a value of 1 may indicate that the QM SEI message contains average quality indicator information for the pictures included in CLVS. Additionally, average quality presence information with a value of 0 may indicate that the QM SEI message does not contain average quality indicator information for the pictures included in CLVS. However, this is not limited to the above, and what the average quality presence information with a value of 1 indicates may be different from what the average quality presence information with a value of 0 indicates.

[0511] Average quality present information may take various forms and may be represented by various names. For example, average quality present information may be a syntax element or a syntax structure containing one or more syntax elements. For example, average quality present information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, average quality present information may include qm_clvs_values_present_flag or qm_clvs_values_present_idc, but is not limited thereto.

[0512] Quality indicator definition information may indicate whether the QM SEI message includes relevant information involved in deriving quality indicators, namely quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality overall reference information, quality indicator length information, quality indicator description existence information, and / or quality indicator description information. For example, quality indicator definition information with a value of 1 may indicate that the QM SEI message includes relevant information involved in deriving quality indicators. Additionally, quality indicator definition information with a value of 0 may indicate that the QM SEI message does not include relevant information involved in deriving quality indicators. However, this is not limited thereto, and what quality indicator definition information with a value of 1 indicates may be different from what quality indicator definition information with a value of 0 indicates.

[0513] Quality metric definition information may take various forms and may be represented by various names. For example, quality metric definition information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality metric definition information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality metric definition information may include qm_metric_definitions_present_flag or qm_metric_definitions_present_idc, but is not limited thereto.

[0514] Quality gain information may indicate whether picture quality indicator information and / or average quality indicator information represents absolute quality or whether picture quality indicator information and / or average quality indicator information represents relative quality relative to a reference object. For example, quality gain information with a value of 1 may indicate that picture quality indicator information and / or average quality indicator information represents relative quality relative to a reference object. Additionally, quality gain information with a value of 0 may indicate that picture quality indicator information and / or average quality indicator information represents absolute quality. However, this is not limited thereto, and what quality gain information with a value of 1 represents may be different from what quality gain information with a value of 0 represents.

[0515] Quality gain information may take various forms and may be represented by various names. For example, quality gain information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality gain information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality gain information may include qm_gain_flag[i] or qm_gain_idc[i], but is not limited thereto.

[0516] Quality gain reference information can be obtained based on the quality gain information indicating that picture quality indicator information and / or average quality indicator information represents relative quality compared to the reference target.

[0517] Quality gain reference information may indicate whether picture quality indicator information and / or average quality indicator information represents relative quality compared to the input image of the current or previous post-processing step, or whether picture quality indicator information and / or average quality indicator information represents relative quality compared to the initial decoded image to which post-processing is not applied. For example, quality gain reference information with a value of 1 may indicate that picture quality indicator information and / or average quality indicator information represents relative quality compared to the initial decoded image to which post-processing is not applied. Additionally, quality gain reference information with a value of 0 may indicate that picture quality indicator information and / or average quality indicator information represents relative quality compared to the input image of the current or previous post-processing step. However, this is not limited thereto, and what quality gain reference information with a value of 1 represents may be different from what quality gain reference information with a value of 0 represents.

[0518] Quality gain reference information may take various forms and may be represented by various names. For example, quality gain reference information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality gain reference information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality gain reference information may include qm_gain_reference_flag[i] or qm_gain_reference_idc[i], but is not limited thereto.

[0519] Quality indicator type information may indicate what type of measurement method the picture quality indicator information and / or average quality indicator information represents. For example, the quality indicator type information may indicate whether the picture quality indicator information and / or average quality indicator information is a user-defined quality indicator, PSNR (Peak Signal-to-Noise Ratio), PSNR-YUV (PSNR integrating Y, U, and V components), SSIM (Structural Similarity), MS-SSIM (Multi-Scale SSIM), MOS (Mean Opinion Score), wPSNR (Weighted PSNR), WS-PSNR (Weighted Spherical PSNR), MSE (Mean Squared Error), or VMAF (Video Multimethod Assessment Fusion). The quality indicator type information is not limited to the types of quality indicators exemplified above and may indicate any unexemplified quality indicator capable of representing the quality of the image.

[0520] Quality metric type information may take various forms and may be represented by various names. For example, quality metric type information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality metric type information may include an unsigned integer consisting of 8 bits or variable-length bits. For example, quality metric type information may include, but is not limited to, qm_metric_type[i], qm_metric_type_flag[i], or qm_metric_type_idc[i].

[0521] Quality three-component information may indicate whether quality indicator information includes quality indicator values ​​for three components. For example, quality three-component information with a value of 1 may indicate that the quality indicator information includes quality indicator values ​​for three components. Quality three-component information with a value of 0 may indicate that the quality indicator information includes a single quality indicator value. However, this is not limited thereto, and what quality three-component information with a value of 1 indicates may be different from what quality three-component information with a value of 0 indicates.

[0522] Quality three-component information may take various forms and may be represented by various names. For example, quality three-component information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality three-component information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality three-component information may include qm_three_component_flag[i] or qm_three_component_idc[i], but is not limited thereto.

[0523] Quality indicator increase information can indicate whether the value of high quality indicator information indicates an improvement in quality. For example, quality indicator increase information with a value of 1 may indicate that the value of high quality indicator information indicates an improvement in quality. Quality indicator increase information with a value of 0 may indicate that the value of low quality indicator information indicates an improvement in quality. However, this is not limited to this, and what quality indicator increase information with a value of 1 indicates may be different from what quality indicator increase information with a value of 0 indicates.

[0524] Quality metric increase information may take various forms and may be represented by various names. For example, quality metric increase information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality metric increase information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality metric increase information may include qm_metric_increasing_flag[i] or qm_metric_increasing_idc[i], but is not limited thereto.

[0525] Quality overall reference information may indicate whether quality indicator information is an overall reference quality indicator obtained by comparing each picture in the list with reference pictures. For example, quality overall reference information with a value of 1 may indicate that the quality indicator information is an overall reference quality indicator obtained by comparing each picture in the list with reference pictures. Quality overall reference information with a value of 0 may indicate that it is not determined whether the quality indicator information is an overall reference quality indicator obtained by comparing each picture in the list with reference pictures. However, this is not limited to, and what quality overall reference information with a value of 1 indicates may change from what quality overall reference information with a value of 0 indicates.

[0526] Quality full reference information may take various forms and may be represented by various names. For example, quality full reference information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality full reference information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality full reference information may include qm_full_reference_flag[i] or qm_full_reference_idc[i], but is not limited thereto.

[0527] Quality indicator length information can represent the length of the quality indicator information in bytes.

[0528] Quality indicator length information may take various forms and may be represented by various names. For example, quality indicator length information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality indicator length information may include an unsigned integer consisting of two bits or variable-length bits. For example, quality total reference information may include qm_value_len_minus1_in_bytes[i], but is not limited thereto.

[0529] Quality indicator description existence information may indicate whether a QM SEI message contains quality indicator description information. For example, quality indicator description existence information with a value of 1 may indicate that the QM SEI message contains quality indicator description information. Additionally, quality indicator description existence information with a value of 0 may indicate that the QM SEI message does not contain quality indicator description information. However, this is not limited thereto, and what quality indicator description existence information with a value of 1 indicates may be different from what quality indicator description existence information with a value of 0 indicates.

[0530] Quality metric description present information may take various forms and may be represented by various names. For example, quality metric description present information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality metric description present information may include a flag consisting of one bit or an indicator consisting of two or more bits or a variable-length bit. For example, quality metric description present information may include qm_metric_description_present_flag[i] or qm_metric_description_present_idc[i], but is not limited thereto.

[0531] Quality indicator description information can display a text description of the quality indicator information.

[0532] Quality metric description information may take various forms and may be expressed by various names. For example, quality metric description information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality metric description information may include a text string composed of variable-length bytes. For example, quality metric description information may include qm_metric_description[i], but is not limited thereto.

[0533] Picture quality indicator information can be obtained based on picture quality existence information indicating the existence of picture quality indicator information.

[0534] Picture quality metric information may represent quality metrics for a single picture. For example, picture quality metric information may represent absolute quality based on the value of quality gain information or relative quality relative to a reference object. For example, picture quality metric information may be a user-defined quality metric based on the value of quality metric type information, or PSNR, PSNR-YUV, SSIM, MS-SSIM, MOS, wPSNR, WS-PSNR, MSE, and / or VMAF.

[0535] Picture quality metric information may take various forms and may be represented by various names. For example, picture quality metric information may be a syntax element or a syntax structure containing one or more syntax elements. For example, picture quality metric information may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, picture quality metric information may include qm_pic_metric_value[i], but is not limited thereto.

[0536] Average quality indicator information can be obtained based on average quality existence information indicating the existence of average quality indicator information.

[0537] Average quality metric information may represent the average quality metric for the pictures included in CLVS. For example, the average quality metric information may represent the absolute average quality based on the value of the quality gain information, or the relative average quality relative to the reference. For example, the average quality metric information may be a user-defined quality metric based on the value of the quality metric type information, or PSNR, PSNR-YUV, SSIM, MS-SSIM, MOS, wPSNR, WS-PSNR, MSE, or VMAF.

[0538] Average quality metric information may take various forms and may be represented by various names. For example, average quality metric information may be a syntax element or a syntax structure containing one or more syntax elements. For example, average quality metric information may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, average quality metric information may include qm_clvs_metric_value[i], but is not limited thereto.

[0539] The decoding device can derive the processing order for SEI messages (S520).

[0540] For example, the processor of the decoding device can derive the processing order for the payload types of SEI messages based on the SPO SEI message.

[0541] SEI messages can have various payload types. For example, an SEI message may include SEI messages having different payload types and / or SEI messages having the same payload type. The processing order of multiple SEI messages can be determined based on these payload types.

[0542] The processor can derive the processing order for SEI messages, including QM SEI messages and PON nested SEI messages, based on the payload type information and processing order information of SPO SEI messages.

[0543] The decoding device can acquire PON-nested SEI messages (S530).

[0544] For example, the processor of the decoding device can obtain PON nested SEI messages based on PON SEI messages.

[0545] Although the drawing illustrates that deriving the processing order of PON-nested SEI messages and acquiring PON-nested SEI messages are performed sequentially, this is merely for the sake of understanding, and the execution order of deriving the processing order of PON-nested SEI messages and acquiring PON-nested SEI messages is not determined by what is shown in the drawing and may be performed independently of each other. In other words, acquiring PON-nested SEI messages may be performed first, followed by deriving the processing order of PON-nested SEI messages.

[0546] The decoding device can obtain quality indicator information for the decoded picture (S540).

[0547] For example, the processor of the decoding device can obtain quality indicators for the decoded picture based on the QM SEI message.

[0548] The processor can obtain quality indicators for a decoded picture based on quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, quality indicator description information, picture quality indicator information and / or average quality indicator information included in the QM SEI message.

[0549] Based on the QM SEI message, the processor can obtain the values ​​of user-defined metrics for the decoded picture, the values ​​of PSNR, PSNR-YUV, SSIM, MS-SSIM, MOS, wPSNR, WS-PSNR, MSE, and / or VMAF.

[0550] Here, picture quality indicator information and / or average quality indicator information correspond to quality indicator information that directly represents the quality indicators for the decoded picture.

[0551] In addition, quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality overall reference information, quality indicator length information, quality indicator description existence information and / or quality indicator description information are involved in obtaining quality indicators for the decoded picture. As such, these information corresponds to relevant information for obtaining quality indicators, in other words, quality indicator-related information.

[0552] Depending on the embodiment, quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, overall quality reference information, quality indicator length information, quality indicator description existence information and / or quality indicator description information may be included in quality indicator related information. Additionally, depending on the embodiment, quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, overall quality reference information, quality indicator length information, quality indicator description existence information and / or quality indicator description information may each correspond to quality indicator related information.

[0553] Here, if the QM SEI message is part of the processing chain of a specific SPO SEI message within CLVS and is the first QM SEI message that is not a PON-nested SEI message, the QM SEI message may contain quality indicator definition information indicating that quality indicator related information exists. For example, if the QM SEI message is part of the processing chain of a specific SPO SEI message within CLVS and is the first QM SEI message that is not a PON-nested SEI, the QM SEI message may contain quality indicator definition information with a value of 1.

[0554] In addition, if a QM SEI message that is part of the processing chain of a specific SPO SEI message and is not a PON-nested SEI message is not the first QM SEI message that is part of the processing chain of the specific SPO SEI message and is not a PON-nested SEI message within CLVS, the QM SEI message corresponds to one of the following two cases.

[0555] 1. A QM SEI message may include quality indicator definition information indicating that quality indicator-related information does not exist. For example, a QM SEI message may include quality indicator definition information with a value of 0.

[0556] 2. A QM SEI message may include information related to quality indicators. In this case, the value of the information related to quality indicators in the QM SEI message is the same as the value of the information related to quality indicators in the first QM SEI message that is part of the processing chain of a specific SPO SEI message and is not a PON-nested SEI message. In other words, the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the QM SEI message are the same as the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the first QM SEI message that is part of the processing chain of the specific SPO SEI message and is not a PON-nested SEI message.

[0557] In other words, based on the fact that a QM SEI message is part of the processing chain of a specific SPO SEI message and is not a PON-nested SEI message, the value of the quality indicator-related information contained in the QM SEI message may be the same as the value of the quality indicator-related information contained in the first QM SEI message that is part of the processing chain of the said specific SPO SEI message and is not a PON-nested SEI message.

[0558] Additionally, based on the fact that the QM SEI message is part of the processing chain of a specific SPO SEI message and is the first QM SEI message that is not a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating the existence of quality indicator-related information.

[0559] Here, if the QM SEI message is the first QM SEI message that is part of the processing chain of a specific SPO SEI message within CLVS and is a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating that quality indicator related information exists. For example, if the QM SEI message is the first QM SEI message that is part of the processing chain of a specific SPO SEI message within CLVS and is a PON-nested SEI, the QM SEI message may include quality indicator definition information with a value of 1.

[0560] Additionally, if a QM SEI message that is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message is not the first QM SEI message that is part of the processing chain of the specific SPO SEI message and is a PON-nested SEI message within CLVS, the QM SEI message corresponds to one of the following two cases.

[0561] 1. A QM SEI message may include quality indicator definition information indicating that quality indicator-related information does not exist. For example, a QM SEI message may include quality indicator definition information with a value of 0.

[0562] 2. A QM SEI message may include information related to quality indicators. In this case, the value of the information related to quality indicators in the QM SEI message is the same as the value of the information related to quality indicators in the first QM SEI message, which is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message. In other words, the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the QM SEI message are the same as the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the first QM SEI message, which is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message.

[0563] In other words, based on the fact that a QM SEI message is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message, the value of the quality indicator-related information contained in the QM SEI message may be the same as the value of the quality indicator-related information contained in the first QM SEI message, which is part of the processing chain of the said specific SPO SEI message and is a PON-nested SEI message.

[0564] Additionally, based on the fact that the QM SEI message is part of the processing chain of a specific SPO SEI message and is the first QM SEI message that is a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating the existence of quality indicator-related information.

[0565] As such, QM SEI messages that are part of the processing chain of a specific SPO SEI message and are not PON-nested SEI messages may have the same quality indicator-related information (e.g., quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information and quality indicator description information) within CLVS, and may have different quality indicators (e.g., picture quality indicator information and / or average quality indicator information).

[0566] Additionally, QM SEI messages, which are part of the processing chain of specific SPO SEI messages and are PON-nested SEI messages, have the same quality indicator-related information (e.g., quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information) within CLVS, and may have different quality indicators (e.g., picture quality indicator information and / or average quality indicator information).

[0567] As a result, QM SEI messages that are not PON nested SEI messages and QM SEI messages that are PON nested SEI messages are clearly distinguished, and it becomes clear that there is no association or dependency between them. In addition, the scope of QM SEI messages that are not PON nested SEI messages (e.g., within CLVS) and the scope of QM SEI messages that are PON nested SEI messages (e.g., within the processing chain and CLVS by the same SPO SEI message) are clarified.

[0568] As a result, malfunctions in the coding system that may occur due to unclear association or dependency between a QM SEI message that is not a PON nested SEI message and a QM SEI message that is a PON nested SEI message, or unclear scope of application between a QM SEI message that is not a PON nested SEI message and a QM SEI message that is a PON nested SEI message, can be suppressed, prevented, or minimized.

[0569] As a result, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0570] In addition, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0571] In addition, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0572] FIG. 6 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0573] The encoding method (S600) may include the operations described below.

[0574] The terms or names described below (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms or names described below. For example, the image information described below may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0575] The operations described below do not constitute an essential component of the decoding method according to one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of the decoding method according to one embodiment, and the previously described operations may be added.

[0576] The sequence of actions illustrated in the drawings regarding the actions to be described below is merely an example, and the actions to be described below may be performed in a different order as long as it does not contradict the causal relationship of the actions to be described.

[0577] The operations described below form a single embodiment integrally with the configurations and / or operations described above, unless they conflict with the configurations and / or operations described above, and do not form a separate embodiment distinct from the configurations and / or operations described above.

[0578] The terms "first," "second," "third," etc., used below are merely distinguishing indicators for identifying specific messages or information among multiple pieces of information or messages that may be included within the CLVS, and are not intended to limit the order, importance, or relative priority of these messages or information, or to restrict them to specific embodiments. For example, "first message (or first information)" refers only to at least one message (or information) included in at least one picture unit within the CLVS, and does not imply that said message (or information) must be encoded or decoded first.

[0579] The encoding method (S600) can be executed by an encoding device including a memory and a processor electrically connected to the memory, for example, by a processor.

[0580] The encoding device can generate QM (Quality Metrics) SEI (Supplemental Enhancement Information) messages (S610).

[0581] SEI messages may convey specific types of information that assist in processes related to the decoding, display, or other purposes of image information. Here, SEI messages may not be necessary for the decoding process to determine the sample values ​​of the decoded picture.

[0582] For example, SEI messages may include SEI processing order (SPO) SEI messages, processing order nesting (PON) SEI messages, and / or Quality Metrics (QM) SEI messages.

[0583] The processor of the encoding device may generate a quality indicator for the decoded picture and generate a QM SEI message based on the quality indicator for the decoded picture. The QM SEI message may include at least one of information regarding the quality of a single picture, information regarding the average quality of all pictures corresponding to the CLVS, information regarding the quality gain of a single picture, and / or information regarding the average quality gain of all pictures corresponding to the CLVS. The information regarding quality may be expressed in various ways, such as quality information, quality indicator information, or quality indicator, and may include subjective or objective indicators regarding the quality of the picture.

[0584] The processor can obtain quality metrics for the decoded picture, including values ​​of user-defined metrics, PSNR (Peak Signal-to-Noise Ratio), PSNR-YUV (PSNR with integrated Y, U, and V components), SSIM (Structural Similarity), MS-SSIM (Multi-Scale SSIM), MOS (Mean Opinion Score), wPSNR (Weighted PSNR), WS-PSNR (Weighted Spherical PSNR), MSE (Mean Squared Error), and / or VMAF (Video Multimethod Assessment Fusion).

[0585] The processor can generate quality indicator information (e.g., picture quality indicator information and / or average quality indicator information) based on quality indicators. Here, the picture quality indicator information and / or average quality indicator information correspond to quality indicator information that directly represents the quality indicators for the decoded picture.

[0586] In addition, the processor may generate relevant information involved in deriving quality indicators, namely quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality overall reference information, quality indicator length information, quality indicator description existence information, and / or quality indicator description information. As such, this information corresponds to relevant information for obtaining quality indicators, in other words, quality indicator related information.

[0587] Depending on the embodiment, quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, overall quality reference information, quality indicator length information, quality indicator description existence information and / or quality indicator description information may be included in quality indicator related information. Additionally, depending on the embodiment, quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, overall quality reference information, quality indicator length information, quality indicator description existence information and / or quality indicator description information may each correspond to quality indicator related information.

[0588] The QM SEI message may be the same as the QM SEI message described in operation 510 of FIG. 5. The description of the QM SEI message may be replaced with the description of the QM SEI message described in operation 510 of FIG. 5.

[0589] The encoding device can generate a PON (Processing order nesting) SEI message (S620).

[0590] For example, the processor of the encoding device may generate a PON SEI message based on a PON nested SEI message. The PON SEI message may include at least one PON nested SEI message in which information regarding the processing order is provided by an SPO SEI message.

[0591] PON nested SEI messages are SEI messages contained within PON SEI messages, and like general SEI messages, they can convey specific types of information that assist in processes related to the decoding, display, or other purposes of image information. If a specific SEI message is not defined in some of the multiple decoding devices, that specific SEI message may be included in the PON SEI message so that undefined SEI messages are ignored by the decoding devices.

[0592] The PON SEI message may be the same as the PON SEI message described in operation 510 of FIG. 5. The description of the PON SEI message may be replaced with the description of the SEI message regarding post-processing described in operation 510 of FIG. 5.

[0593] The encoding device can generate an SPO (SEI processing order) SEI message (S630).

[0594] For example, a processor of an encoding device can determine the processing order for a plurality of SEI messages, including QM SEI messages and PON-nested SEI messages. The processor can determine the processing order for a plurality of SEI messages, including QM SEI messages and PON-nested SEI messages, based on the payload types of the plurality of SEI messages. In other words, the processor can determine the processing order based on the payload types of the plurality of SEI messages.

[0595] SEI messages can have various payload types. For example, an SEI message may include SEI messages having different payload types and / or SEI messages having the same payload type. The processing order of multiple SEI messages can be determined based on these payload types.

[0596] The processor can generate payload type information based on the payload types of multiple SEI messages and generate processing order information based on the processing order of the payload types. The processor can generate an SPO SEI message containing payload type information and processing order information.

[0597] The SPO SEI message may be the same as the SPO SEI message described in operation 510 of FIG. 5. The description of the SPO SEI message may be replaced with the description of the SPO SEI message described in operation 510 of FIG. 5.

[0598] Although the drawing illustrates that generating PON SEI messages and determining the processing order of PON nested SEI messages are performed sequentially, this is merely for illustrative purposes, and the execution order of generating PON SEI messages and determining the processing order of PON nested SEI messages is not determined by what is shown in the drawing and may be performed independently of each other. In other words, determining the processing order of PON nested SEI messages may be performed first, followed by generating PON SEI messages.

[0599] The encoding device can encode video information (S640).

[0600] For example, the processor of the encoding device can encode image information including QM SEI messages, PON SEI messages, and / or SPO SEI messages.

[0601] Video information encoded according to the encoding method (S600) described above can be output in the form of a bitstream. In other words, the bitstream can be generated based on the video information encoded according to the encoding method (S600) described above.

[0602] A bitstream generated based on video information encoded according to the encoding method (S600) described above can be stored on a computer-readable storage medium.

[0603] A bitstream generated based on video information encoded according to the encoding method (S600) described above can be transmitted through a transmission unit and / or a transmission medium.

[0604] Here, if the QM SEI message is part of the processing chain of the same SPO SEI message within the CLVS and is the first QM SEI message that is not a PON-nested SEI message, the QM SEI message may contain quality indicator definition information indicating that quality indicator related information exists. For example, if the QM SEI message is part of the processing chain of a specific SPO SEI message within the CLVS and is the first QM SEI message that is not a PON-nested SEI, the QM SEI message may contain quality indicator definition information with a value of 1.

[0605] In addition, if a QM SEI message that is part of the processing chain of a specific SPO SEI message and is not a PON-nested SEI message is not the first QM SEI message that is part of the processing chain of the specific SPO SEI message and is not a PON-nested SEI message within CLVS, the QM SEI message corresponds to one of the following two cases.

[0606] 1. A QM SEI message may include quality indicator definition information indicating that quality indicator-related information does not exist. For example, a QM SEI message may include quality indicator definition information with a value of 0.

[0607] 2. A QM SEI message may include information related to quality indicators. In this case, the value of the information related to quality indicators in the QM SEI message is the same as the value of the information related to quality indicators in the first QM SEI message that is part of the processing chain of a specific SPO SEI message and is not a PON-nested SEI message. In other words, the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the QM SEI message are the same as the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the first QM SEI message that is part of the processing chain of the specific SPO SEI message and is not a PON-nested SEI message.

[0608] In other words, based on the fact that a QM SEI message is part of the processing chain of a specific SPO SEI message and is not a PON-nested SEI message, the value of the quality indicator-related information contained in the QM SEI message may be the same as the value of the quality indicator-related information contained in the first QM SEI message that is part of the processing chain of the said specific SPO SEI message and is not a PON-nested SEI message.

[0609] Additionally, based on the fact that the QM SEI message is part of the processing chain of a specific SPO SEI message and is the first QM SEI message that is not a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating the existence of quality indicator-related information.

[0610] Here, if the QM SEI message is the first QM SEI message that is part of the processing chain of a specific SPO SEI message within CLVS and is a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating that quality indicator related information exists. For example, if the QM SEI message is the first QM SEI message that is part of the processing chain of a specific SPO SEI message within CLVS and is a PON-nested SEI, the QM SEI message may include quality indicator definition information with a value of 1.

[0611] Additionally, if a QM SEI message that is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message is not the first QM SEI message that is part of the processing chain of the specific SPO SEI message and is a PON-nested SEI message within CLVS, the QM SEI message corresponds to one of the following two cases.

[0612] 1. A QM SEI message may include quality indicator definition information indicating that quality indicator-related information does not exist. For example, a QM SEI message may include quality indicator definition information with a value of 0.

[0613] 2. A QM SEI message may include information related to quality indicators. In this case, the value of the information related to quality indicators in the QM SEI message is the same as the value of the information related to quality indicators in the first QM SEI message, which is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message. In other words, the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the QM SEI message are the same as the respective values ​​of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information included in the first QM SEI message, which is part of the processing chain of the specific SPO SEI message and is a PON-nested SEI message.

[0614] In other words, based on the fact that a QM SEI message is part of the processing chain of a specific SPO SEI message and is a PON-nested SEI message, the value of the quality indicator-related information contained in the QM SEI message may be the same as the value of the quality indicator-related information contained in the first QM SEI message, which is part of the processing chain of the said specific SPO SEI message and is a PON-nested SEI message.

[0615] Additionally, based on the fact that the QM SEI message is part of the processing chain of a specific SPO SEI message and is the first QM SEI message that is a PON-nested SEI message, the QM SEI message may include quality indicator definition information indicating the existence of quality indicator-related information.

[0616] As such, QM SEI messages that are part of the processing chain of a specific SPO SEI message and are not PON-nested SEI messages may have the same quality indicator-related information (e.g., quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information and quality indicator description information) within CLVS, and may have different quality indicators (e.g., picture quality indicator information and / or average quality indicator information).

[0617] Additionally, QM SEI messages, which are part of the processing chain of specific SPO SEI messages and are PON-nested SEI messages, have the same quality indicator-related information (e.g., quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality total reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information) within CLVS, and may have different quality indicators (e.g., picture quality indicator information and / or average quality indicator information).

[0618] As a result, QM SEI messages that are not PON nested SEI messages and QM SEI messages that are PON nested SEI messages are clearly distinguished, and it becomes clear that there is no association or dependency between them. In addition, the scope of QM SEI messages that are not PON nested SEI messages (e.g., within CLVS) and the scope of QM SEI messages that are PON nested SEI messages (e.g., within the processing chain and CLVS by the same SPO SEI message) are clarified.

[0619] As a result, malfunctions in the coding system that may occur due to unclear association or dependency between a QM SEI message that is not a PON nested SEI message and a QM SEI message that is a PON nested SEI message, or unclear scope of application between a QM SEI message that is not a PON nested SEI message and a QM SEI message that is a PON nested SEI message, can be suppressed, prevented, or minimized.

[0620] As a result, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0621] In addition, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0622] In addition, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0623] FIG. 7 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0624] As illustrated in FIG. 7, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0625] The above encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data to generate a bitstream and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.

[0626] The bitstream may be generated by a video encoding method and / or encoding device to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0627] The streaming server transmits multimedia data to a user device based on a user request through a web server, and the web server can act as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server can perform the role of controlling commands and responses between each device within the content streaming system.

[0628] The streaming server can receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store the bitstream for a certain period of time.

[0629] Examples of the above user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0630] Each server within the above-mentioned content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0631] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.

[0632] An embodiment according to the present disclosure can be used to encode / decode images.

Claims

1. Regarding the method of decoding video information, Acquire SPO (sei processing order) SEI (Supplemental Enhancement Information) messages, PON (Processing order nesting) SEI messages, and QM (Quality Metrics) SEI messages; Deriving the processing order of at least one SEI message based on the above SPO SEI message; Based on the above PON SEI message, at least one PON nested SEI message is obtained; Includes obtaining quality indicators based on the above QM SEI message, A method in which, based on the fact that the above QM SEI message is part of the processing chain of the above SPO SEI message and is a PON-nested SEI message, the value of the quality indicator-related information included in the above QM SEI message is the same as the value of the quality indicator-related information included in the first QM SEI message, which is part of the processing chain of the above SPO SEI message and is a PON-nested SEI message.

2. In Paragraph 1, A method in which the above QM SEI message is part of the processing chain of the above SPO SEI message and is the first QM SEI message that is a PON nested SEI message, and the above QM SEI message includes quality indicator definition information indicating that the above quality indicator related information exists.

3. In Paragraph 1, A method in which the value of the quality indicator related information of the above QM SEI message is the same as the value of the quality indicator related information of the first QM SEI message that is part of the processing chain of the above SPO SEI message and is not a PON-nested SEI message.

4. In Paragraph 1, A method in which the above QM SEI message is part of the processing chain of the above SPO SEI message and is the first QM SEI message that is not a PON nested SEI message, and the above QM SEI message includes quality indicator definition information indicating that the above quality indicator related information exists.

5. In Paragraph 1, A method in which the above quality indicator-related information is any one of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality overall reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information.

6. Regarding the method of encoding video information, Generate QM (Quality Metrics) SEI (Supplemental Enhancement Information) messages based on quality indicators; Generate a PON (Processing order nesting) SEI message based on at least one PON nested SEI message; Generate an SPO (Sei processing order) SEI message based on the processing order of at least one SEI message including at least one PON nested SEI message and the QM SEI message; Encoding the above QM SEI message, the above PON SEI message, and the above SPO SEI message, etc. A method in which, based on the fact that the above QM SEI message is part of the processing chain of the above SPO SEI message and is a PON-nested SEI message, the value of the quality indicator-related information included in the above QM SEI message is the same as the value of the quality indicator-related information included in the first QM SEI message, which is part of the processing chain of the above SPO SEI message and is a PON-nested SEI message.

7. In Paragraph 6, A method in which the above QM SEI message is part of the processing chain of the above SPO SEI message and is the first QM SEI message that is a PON nested SEI message, and the above QM SEI message includes quality indicator definition information indicating that the above quality indicator related information exists.

8. In Paragraph 6, A method in which the value of the quality indicator related information of the above QM SEI message is the same as the value of the quality indicator related information of the first QM SEI message that is part of the processing chain of the above SPO SEI message and is not a PON-nested SEI message.

9. In Paragraph 6, A method in which the above QM SEI message is part of the processing chain of the above SPO SEI message and is the first QM SEI message that is not a PON nested SEI message, and the above QM SEI message includes quality indicator definition information indicating that the above quality indicator related information exists.

10. In Paragraph 6, A method in which the above quality indicator-related information is any one of quality gain information, quality gain reference information, quality indicator type information, quality three-component information, quality indicator increase information, quality overall reference information, quality indicator length information, quality indicator description existence information, and quality indicator description information.

11. A computer-readable storage medium for storing a bitstream generated based on the method of paragraph 6.

12. Regarding methods concerning bitstreams, Generate the above bitstream; Includes transmitting data including the above bitstream, The above bitstream is generated based on generating a QM (Quality Metrics) SEI (Supplemental Enhancement Information) message based on quality metric information, generating a PON (Processing Order Nesting) SEI message based on at least one PON-nested SEI message, generating an SPO (Sei Processing Order) SEI message based on processing order information of at least one SEI message including the at least one PON-nested SEI message and the QM SEI message, and encoding the QM SEI message, the PON SEI message, and the SPO SEI message. A method in which, based on the fact that the above QM SEI message is part of the processing chain of the above SPO SEI message and is a PON-nested SEI message, the value of the quality indicator-related information included in the above QM SEI message is the same as the value of the quality indicator-related information included in the first QM SEI message, which is part of the processing chain of the above SPO SEI message and is a PON-nested SEI message.

Images