Method for decoding video information, method for encoding video information, method related to bitstream, and computer-readable storage medium

Abstract

A video decoding method performed by a decoding device, according to the present disclosure, comprises the steps of: acquiring a quality metric (QM) supplemental enhancement information (SEI) message included in at least one picture unit; and acquiring quality metric information on the basis of the QM SEI message, wherein the quality metric information includes information on at least one sub-layer for the picture unit, and average quality metric information corresponding to the at least one sub-layer.

Description

Method for decoding image information, method for encoding image information, method relating to a bitstream and a computer-readable storage medium

[0001] The present disclosure relates to a method for decoding image information, a method for encoding image information, a method for bitstreams, and a computer-readable storage medium.

[0002] Recently, the demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition), has been increasing across various fields. As video data becomes higher in resolution and quality, the relative amount of information or bits transmitted increases compared to conventional video data. This increase in the amount of transmitted information or bits leads to higher transmission and storage costs.

[0003] Accordingly, high-efficiency video compression technology is required to effectively transmit, store, and play back high-resolution, high-quality video information.

[0004] The present disclosure aims to improve the reliability of a coding system including an encoding device and a decoding device.

[0005] The present disclosure aims to improve the coding efficiency of a coding system including an encoding device and a decoding device.

[0006] The present disclosure aims to improve the data transmission efficiency of a coding system including an encoding device and a decoding device.

[0007] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0008] According to one aspect of the present disclosure, an image decoding method comprises the steps of: acquiring a quality metric (QM) supplemental enhancement information (SEI) message included in at least one picture unit; and acquiring quality indicator information based on the QM SEI message, wherein the quality indicator information includes information regarding at least one sub-layer for the picture unit and average quality indicator information corresponding to the at least one sub-layer.

[0009] According to one aspect of the present disclosure, a decoding device for decoding image information acquires a quality metric (QM) supplemental enhancement information (SEI) message included in at least one picture unit, acquires quality indicator information based on the QM SEI message, and the quality indicator information includes information regarding at least one sub-layer for the picture unit and average quality indicator information corresponding to the at least one sub-layer.

[0010] In a method or device for decoding the above image information, the information regarding the at least one sublayer may be characterized by including information on the maximum number of the at least one sublayer and information on the number of pictures included in the at least one sublayer.

[0011] In a method or device for decoding the above image information, the picture count information included in the at least one sublayer may be characterized by representing at least one picture count having an identifier smaller than or equal to the identifier corresponding to the sublayer.

[0012] In a method or device for decoding the above image information, the average quality indicator information may be characterized by being derived by averaging the quality indicator values ​​of at least one picture having an identifier smaller than or equal to the identifier corresponding to the sublayer, based on picture count information included in the at least one sublayer.

[0013] In a method or device for decoding the above image information, the picture count information included in the at least one sublayer may be characterized by being obtained based on the number of sublayers according to the maximum count information.

[0014] In a method or device for decoding the above image information, the quality indicator information further includes information on whether average quality exists, and based on the information on whether average quality exists, the maximum number information and the picture number information may be obtained.

[0015] According to one aspect of the present disclosure, an image encoding method comprises: generating at least one Quality Metic (QM) supplemental enhancement information (SEI) message included in at least one picture unit; and encoding image information including the at least one QM SEI message, wherein the quality indicator information includes information regarding at least one sub-layer for the picture unit and average quality indicator information corresponding to the at least one sub-layer.

[0016] According to one aspect of the present disclosure, an apparatus for encoding image information generates at least one Quality Metic (QM) supplemental enhancement information (SEI) message included in at least one picture unit, encodes image information including the at least one QM SEI message, and the quality indicator information comprises information regarding at least one sub-layer for the picture unit and average quality indicator information corresponding to the at least one sub-layer.

[0017] In a method or device for encoding the above-mentioned image information, the information regarding the at least one sublayer may be characterized by including information on the maximum number of the at least one sublayer and information on the number of pictures included in the at least one sublayer.

[0018] In a method or device for encoding the above-mentioned image information, the picture count information included in the at least one sublayer may be characterized by representing at least one picture count having an identifier smaller than or equal to the identifier corresponding to the sublayer.

[0019] In a method or device for encoding the above-mentioned image information, the average quality indicator information may be characterized by being generated by averaging the quality indicator values ​​of at least one picture having an identifier smaller than or equal to the identifier corresponding to the at least one sublayer, based on picture count information included in the at least one sublayer.

[0020] In a method or device for encoding the above-mentioned image information, the picture count information included in the at least one sublayer may be characterized by being generated based on the number of sublayers according to the maximum count information.

[0021] In a method or device for encoding the above image information, the maximum number information and the picture number information may be generated based on the average quality indicator information existing within the QM SEI message.

[0022] According to one aspect of the present disclosure, a bitstream generated by an image encoding method is stored in a computer-readable storage medium. At least one Quality Metic (QM) supplemental enhancement information (SEI) message included in at least one picture unit is generated based on quality indicator information, and a bitstream generated based on image information including the at least one QM SEI message is stored in a computer-readable storage medium. The quality indicator information is characterized by including information regarding at least one sub-layer for the picture unit and average quality indicator information corresponding to the at least one sub-layer.

[0023] According to one aspect of the present disclosure, a method for transmitting data for an image comprises: a step of acquiring image information, wherein the image information includes at least one Quality Metic (QM) SEI (supplemental enhancement information) message included in at least one picture unit; and a step of transmitting the data including the image information, wherein the quality indicator information includes information regarding at least one sub-layer for the picture unit and average quality indicator information corresponding to the at least one sub-layer.

[0024] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0025] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0026] According to the present disclosure, the reliability of a coding system including an encoding device and a decoding device can be improved.

[0027] According to the present disclosure, the coding efficiency of a coding system including an encoding device and a decoding device can be improved.

[0028] According to the present disclosure, the data transmission efficiency of a coding system including an encoding device and a decoding device can be improved.

[0029] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0030] FIG. 1 is a schematic diagram illustrating a video coding system to which an embodiment according to the present disclosure can be applied.

[0031] FIG. 2 is a schematic diagram showing an encoding device to which an embodiment according to the present disclosure can be applied.

[0032] FIG. 3 is a schematic diagram showing a decoding device to which an embodiment according to the present disclosure can be applied.

[0033] Figure 4 illustrates an exemplary hierarchical structure for a coded video / image.

[0034] FIG. 5 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0035] FIG. 6 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0036] FIG. 7 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0037] Hereinafter, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0038] In describing the embodiments of the present disclosure, detailed descriptions of known configurations or functions are omitted if it is determined that such descriptions could obscure the essence of the present disclosure. Additionally, parts of the drawings unrelated to the description of the present disclosure have been omitted, and similar parts are denoted by similar reference numerals.

[0039] In the present disclosure, when a component is described as being "connected," "combined," or "joined" with another component, this may include not only a direct connection but also an indirect connection in which another component exists in between. Furthermore, when a component is described as "comprising" or "having" another component, this means that, unless specifically stated otherwise, it does not exclude the other component but may include an additional component.

[0040] In the present disclosure, terms such as first, second, etc. are used solely for the purpose of distinguishing one component from another and do not limit the order or importance of the components unless specifically stated otherwise. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and likewise, a second component in one embodiment may be referred to as a first component in another embodiment.

[0041] In this disclosure, distinct components are intended to clearly describe their respective features and do not imply that the components are separate. That is, multiple components may be integrated to form a single hardware or software unit, or a single component may be distributed to form multiple hardware or software units. Accordingly, such integrated or distributed embodiments are included within the scope of this disclosure, unless otherwise noted.

[0042] In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present disclosure. Furthermore, embodiments including additional components in addition to the components described in various embodiments are also included within the scope of the present disclosure.

[0043] The present disclosure relates to the encoding and decoding of images. For example, the methods and embodiments disclosed in this document may be applied to methods disclosed in the VVC (versatile video coding) standard, EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard) or next-generation video / image coding standards (e.g., H.267 or H.268).

[0044] The present disclosure presents various embodiments relating to video / image coding, and unless otherwise stated, said embodiments may be performed in combination with one another.

[0045] Unless newly defined in this disclosure, the terms used herein may have the ordinary meanings commonly used in the technical field to which this disclosure belongs.

[0046] In this disclosure, "video" may refer to a set of images over time. In this disclosure, "picture" generally refers to a unit representing a single image at a specific time, and a slice / tile is a unit that constitutes a part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices / tiles. A picture may be composed of one or more tile groups. A tile group may include one or more tiles. A brick may represent a rectangular area of ​​rows of CTUs within a tile in a picture. In this document, tile groups and slices may be used interchangeably. For example, in this document, a tile group / tile group header may be referred to as a slice / slice header.

[0047] In the present disclosure, "pixel" or "pel" may refer to the smallest unit constituting a picture (or image). Additionally, "sample" may be used as a term corresponding to pixel. A sample may generally represent a pixel or a pixel value, may represent only the pixel / pixel value of the luminance component, or may represent only the pixel / pixel value of the chroma component.

[0048] In this disclosure, "unit" may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. A unit may include one luminance block and two chroma (e.g., cb, cr) blocks. Depending on the case, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0049] In the present disclosure, "current block" may mean one of "current coding block," "current coding unit," "block to be encoded," "block to be decoded," or "block to be processed." When prediction is performed, "current block" may mean "current prediction block" or "block to be predicted." When transformation (inverse transformation) / quantization (inverse quantization) is performed, "current block" may mean "current transformation block" or "block to be transformed." When filtering is performed, "current block" may mean "block to be filtered."

[0050] In the present disclosure, "current block" may mean a block comprising both a luminous component block and a chroma component block, or "luma block of the current block," unless explicitly stated as a chroma block. The luminous component block of the current block may be expressed by including an explicit description of a luminous component block, such as "luma block" or "current luminous block." Additionally, the chroma component block of the current block may be expressed by including an explicit description of a chroma component block, such as "chroma block" or "current chroma block."

[0051] In the present disclosure, " / " and "," may be interpreted as "and / or." For example, "A / B" and "A, B" may be interpreted as "A and / or B." Additionally, "A / B / C" and "A, B, C" may mean "at least one of A, B and / or c."

[0052] In the present disclosure, "or" may be interpreted as "and / or". For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B". Alternatively, in the present disclosure, "or" may mean "additionally or alternatively".

[0053] FIG. 1 is a schematic diagram illustrating a video / image coding system to which an embodiment according to the present disclosure can be applied.

[0054] Referring to FIG. 1, a video / image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image or data in the form of a file or streaming to the receiving device via a digital storage medium or a network.

[0055] The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be called a video / image encoding device, and the decoding device may be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, and the display unit may be composed of a separate device or an external component.

[0056] A video source may acquire video / images through processes such as video / image capture, synthesis, or generation. The video source may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, etc., and may generate video / images (electronically). For example, virtual video / images may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process in which related data is generated.

[0057] The encoding device can encode input video / images. The encoding device can perform a series of procedures, such as prediction, transformation, and quantization, for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0058] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0059] The decoding device can decode video / images by performing a series of procedures such as inverse quantization, inverse transform, and prediction corresponding to the operation of the encoding device.

[0060] The renderer can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0061] FIG. 2 is a schematic diagram illustrating an encoding device to which an embodiment according to the present disclosure can be applied.

[0062] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-predictor (221) and an intra-predictor (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit (250) may be referred to as a reconstructor or a reconstructed block generator. The above-described image segmentation unit (210), prediction unit (220), residual processing unit (230), entropy encoding unit (240), addition unit (250), and filtering unit (260) may be configured by one or more hardware components (e.g., an encoder chipset or processor) according to the embodiment. Additionally, the memory (270) may include a DPB (Decoded Picture Buffer) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0063] The image segmentation unit (210) can divide an input image (or picture, frame) input to an encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). A coding unit may be recursively divided into a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, a single coding unit may be divided into multiple coding units of a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary-tree structure. For example, a quad-tree structure may be applied first, and a binary-tree structure and / or a ternary-tree structure may be applied later. Alternatively, a binary-tree structure may be applied first. A coding procedure according to the present disclosure may be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the maximum coding unit may be recursively divided into lower-depth coding units so that a coding unit of the optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may each be divided or partitioned from the final coding unit.The above prediction unit may be a unit of sample prediction, and the above transformation unit may be a unit that derives transformation coefficients and / or a unit that derives a residual signal from transformation coefficients.

[0064] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.

[0065] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, as illustrated, the unit that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231). The prediction unit (220) can perform a prediction for a block to be processed (hereinafter, current block) and generate a predicted block (predicted block) containing prediction samples for said current block. The prediction unit (220) can determine whether intra prediction is applied or inter prediction is applied in units of the current block or CU. The prediction unit (220) can generate various information regarding prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (240). The information regarding prediction can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0066] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a Planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, more or fewer directional prediction modes may be used. The intra prediction unit (222) may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0067] The inter prediction unit (221) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different from each other. The temporal neighboring blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc. A reference picture containing the aforementioned temporal surrounding blocks may be called a collocated picture (colPic). For example, the inter prediction unit (221) may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of surrounding blocks as motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0068] The prediction unit (220) may generate a prediction signal based on various prediction methods and / or prediction techniques described below. For example, the prediction unit (220) may apply intra prediction or inter prediction for the prediction of the current block, as well as apply intra prediction and inter prediction simultaneously. A prediction method that applies intra prediction and inter prediction simultaneously for the prediction of the current block may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit (220) may be based on an intra block copy (IBC) prediction mode or a palette mode for the prediction of the block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. Palette mode can be viewed as an example of intra-coding or intra-prediction. When palette mode is applied, sample values ​​within a picture can be signaled based on information regarding palette tables and palette indices.

[0069] The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or to generate a residual signal. The subtraction unit (231) can generate a residual signal (residual signal, residual block, residual sample array) by subtracting the prediction signal (predicted block, prediction sample array) output from the prediction unit (220) from the input image signal (original block, original sample array). The generated residual signal can be transmitted to the conversion unit (232).

[0070] The transformation unit (232) can generate transform coefficients by applying a transformation technique to a residual signal. For example, the transformation technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transformation obtained based on a prediction signal generated using all previously reconstructed pixels. The transformation process may be applied to a block of pixels of the same size in a square, or to a block of variable size that is not square.

[0071] The quantization unit (233) can quantize the transformation coefficients and transmit them to the entropy encoding unit (240). The entropy encoding unit (240) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (233) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients.

[0072] The entropy encoding unit (240) can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (190) may encode information required for video / image restoration (e.g., values ​​of syntax elements) together or separately, in addition to quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The signaling information, transmitted information, and / or syntax elements mentioned in the present disclosure may be included in the video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream.

[0073] The above bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) for transmitting a signal output from the entropy encoding unit (240) and / or a storage unit (not shown) for storing it may be provided as an internal / external element of the encoding device (200), or the transmission unit may be provided as a component of the entropy encoding unit (240).

[0074] The quantized transformation coefficients output from the quantization unit (233) can be used to generate a residual signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit (234) and the inverse transformation unit (235).

[0075] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0076] The adder (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit (221) or the intra prediction unit (222). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (250) may be called a reconstructed unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after undergoing filtering as described below.

[0077] The filtering unit (260) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (260) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (270), specifically in the DPB of memory (170). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240), as described below in the description of each filtering method. The information regarding filtering can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0078] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter-prediction unit (221). Through this, the encoding device (200) can avoid prediction mismatches between the encoding device (200) and the decoding device when inter-prediction is applied, and can also improve encoding efficiency.

[0079] The DPB in memory (270) can store a modified restored picture to be used as a reference picture in the inter prediction unit (221). Memory (270) can store motion information of blocks from which motion information is derived (or encoded) in the current picture and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter prediction unit (221) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory (270) can store restoration samples of restored blocks in the current picture and transmit them to the intra prediction unit (222).

[0080] FIG. 3 is a schematic diagram illustrating a decoding device to which an embodiment according to the present disclosure can be applied.

[0081] As illustrated in FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-predictor (332) and an intra-predictor (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321). The aforementioned entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) may be configured by a single hardware component (e.g., a decoder chipset or a processor) according to an embodiment. Additionally, the memory (360) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0082] When a bitstream containing video / image information is input, the decoding device (300) can restore the image by performing a process corresponding to the process performed by the encoding device (200) of FIG. 2. For example, the decoding device (300) can perform decoding using a processing unit applied in the encoding device (200). Thus, the processing unit for decoding may be, for example, a coding unit. The coding unit may be a coding tree unit, or a maximum coding unit may be obtained by dividing it according to a quad tree structure, a binary tree structure, and / or a binary tree structure. And, the restored image signal decoded and output through the decoding device (300) can be played back through a playback device (not shown).

[0083] The decoding device (300) can receive a signal output from the encoding device (200) of FIG. 2 in the form of a bitstream. The received signal can be decoded through an entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The decoding device (300) can decode the picture based on the information regarding the parameter sets and / or the general constraint information. The signaling / received information and / or syntax elements described below can be obtained from the bitstream by decoding through the decoding procedure. For example, the entropy decoding unit (310) can decode information within the bitstream based on coding methods such as exponential coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine a context model using information on the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information on symbols / bins decoded in the previous step, predict the probability of occurrence of the bin according to the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin after determining the context model. Among the information decoded in the entropy decoding unit (310), information regarding prediction is provided to the prediction unit (330), and residual values ​​for which entropy decoding was performed in the entropy decoding unit (310), i.e., quantized transformation coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive residual signals (residual blocks, residual samples, residual sample array). Additionally, among the information decoded in the entropy decoding unit (310), information regarding filtering can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of the entropy decoding unit (310). Meanwhile, the decoding device according to the present document may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit (310), and the sample decoder may include at least one of the inverse quantization unit (321), inverse transform unit (322), adder (340), filtering unit (350), memory (360), inter prediction unit (332), and intra prediction unit (331).

[0084] In the inverse quantization unit (321), the quantized transformation coefficients can be inversely quantized to output transformation coefficients. The inverse quantization unit (321) can rearrange the quantized transformation coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device (200). The inverse quantization unit (321) can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain transformation coefficients.

[0085] In the inverse conversion unit (322), the conversion coefficients can be inversely converted to obtain a residual signal (residual block, residual sample array).

[0086] The prediction unit (330) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, for example, screen content coding (SCC). IBC basically performs prediction within the current picture, but it can be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described in this document. The palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, information regarding the palette table and palette index can be included in the above video / image information and signaled.

[0087] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The description of the intra prediction unit (222) may be applied equally to the intra prediction unit (331). The referenced samples may be located in the neighborhood of the current block or located away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit (331) may determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0088] The inter prediction unit (332) can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) may construct a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-prediction can be performed based on various prediction modes (techniques), and information regarding the prediction may include information indicating the mode (technique) of inter-prediction for the current block.

[0089] The adder (340) can generate a restoration signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the prediction signal (predicted block, predicted sample array) output from the prediction unit (330) (including the inter prediction unit (332) and / or intra prediction unit (331)). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the restoration block. The description of the adder (250) can be applied equally to the adder (340). The adder (340) may be called a restoration unit or a restoration block generation unit. The generated restoration signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after undergoing filtering as described below.

[0090] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0091] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (360), specifically in the DPB of memory (360). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0092] The (modified) restored picture stored in the DPB of the memory (360) can be used as a reference picture in the inter-prediction unit (332). The memory (360) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (332) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (360) can store restoration samples of blocks restored within the current picture and transmit them to the intra-prediction unit (331).

[0093] In this specification, the embodiments described in the filtering unit (260), inter prediction unit (221), and intra prediction unit (222) of the encoding device (200) may be applied to the filtering unit (350), inter prediction unit (332), and intra prediction unit (331) of the decoding device (300) in the same or corresponding manner.

[0094] Figure 4 illustrates an exemplary hierarchical structure for a coded video / image.

[0095] Referring to Figure 4, the coded image is divided into a VCL (video coding layer) that handles the decoding processing of the image and the image itself, a subsystem that transmits and stores the encoded information, and a NAL (network abstraction layer) that exists between the VCL and the subsystem and is responsible for network adaptation functions.

[0096] In VCL, VCL data containing compressed image data (slice data) can be generated, or parameter sets containing information such as Picture Parameter Set (PPS), Sequence Parameter Set (SPS), and Video Parameter Set (VPS), or SEI (Supplemental Enhancement Information) messages that are additionally required in the decoding process of the image can be generated.

[0097] In NAL, a NAL unit can be created by adding header information (NAL unit header) to the Raw Byte Sequence Payload (RBSP) generated in VCL. In this case, the RBSP refers to slice data, parameter sets, SEI messages, etc. generated in VCL. The NAL unit header may include NAL unit type information specified according to the RBSP data included in the NAL unit.

[0098] As shown in FIG. 4, NAL units can be classified into VCL NAL units and Non-VCL NAL units depending on the RBSP generated in VCL. A VCL NAL unit may refer to a NAL unit containing information about an image (slice data), and a Non-VCL NAL unit may refer to a NAL unit containing information necessary to decode an image (parameter set or SEI message).

[0099] The aforementioned VCL NAL unit and Non-VCL NAL unit can be transmitted over a network by attaching header information according to the data specifications of the underlying system. For example, the NAL unit can be transformed into a data format of a specified specification, such as H.266 / VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

[0100] As described above, the NAL unit type can be determined according to the RBSP data structure included in the NAL unit, and information about this NAL unit type can be stored in the NAL unit header and signaled.

[0101] For example, NAL units can be broadly classified into VCL NAL unit types and Non-VCL NAL unit types depending on whether they contain information about the image (slice data). VCL NAL unit types can be classified according to the properties and types of the picture included in the VCL NAL unit, while Non-VCL NAL unit types can be classified according to the types of parameter sets.

[0102] The following is an example of a NAL unit type specified according to the type of parameter set included in the Non-VCL NAL unit type.

[0103] - APS (Adaptation Parameter Set) NAL unit: Type for the NAL unit containing the APS

[0104] - DPS (Decoding Parameter Set) NAL unit: Type for the NAL unit containing the DPS

[0105] - VPS (Video Parameter Set) NAL unit: Type for the NAL unit containing the VPS

[0106] - SPS (Sequence Parameter Set) NAL unit: Type for the NAL unit containing the SPS

[0107] - PPS(Picture Parameter Set) NAL unit: Type for the NAL unit containing the PPS

[0108] The above-described NAL unit types have syntax information for the NAL unit type, and said syntax information can be stored in the NAL unit header and signaled. For example, said syntax information may be nal_unit_type, and NAL unit types may be specified by the nal_unit_type value.

[0109] The slice header (slice header syntax) may include information / parameters that can be commonly applied to the slice. The APS (APS syntax) or PPS (PPS syntax) may include information / parameters that can be commonly applied to one or more slices or pictures. The SPS (SPS syntax) may include information / parameters that can be commonly applied to one or more sequences. The VPS (VPS syntax) may include information / parameters that can be commonly applied to multiple layers. The DPS (DPS syntax) may include information / parameters that can be commonly applied to the entire video. The DPS may include information / parameters related to the concatenation of the CVS (coded video sequence). In this document, the term High level syntax (HLS) may include at least one of the above APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, and slice header syntax.

[0110] In the present disclosure, image / video information encoded from an encoding device to a decoding device and signaled in the form of a bitstream includes not only information related to partitioning within a picture, intra / inter prediction information, residual information, in-loop filtering information, etc., but may also include information included in the slice header, information included in the APS, information included in the PPS, information included in the SPS, and / or information included in the VPS.

[0111] The following descriptor of the present disclosure specifies the parsing process for each syntax element.

[0112] - ae(v) is a context-adaptive arithmetic entropy-coded syntax element.

[0113] - b(8) is a byte with an arbitrary bit string pattern of length 8 bits. The parsing process of this descriptor is defined by the return value of the function read_bits(8).

[0114] - f(n) is an n-bit fixed-pattern bit string where the left bits are written first. The parsing process of this descriptor is defined by the return value of the function read_bits(n).

[0115] - i(n) is a signed integer using n bits, where n is denoted as "v" in the syntax table, the number of bits varies depending on the values ​​of other syntax elements. The parsing process of this descriptor is defined by interpreting the return value of the function read_bits(n) into a two's complement integer representation where the most significant bit is written first.

[0116] - se(v) is a 0th-order Exp-Golomb encoded signed integer syntax element, written starting from the left bit. The parsing process of this descriptor is defined with order k set to 0.

[0117] - st(v) is a null-terminated string encoded in UCS (Universal Coded Character Set) transmission format 8 (UTF-8) characters as defined in ISO / IEC 10646. The parsing process is defined as follows.

[0118] st(v) starts at a byte-aligned position within the bitstream and reads and returns a sequence of consecutive bytes from the bitstream, starting from the current position and excluding the next byte-aligned byte with a value of 0x00. The bitstream pointer then advances by (stringLength + 1) * 8 bit positions, where stringLength is equal to the number of returned bytes.

[0119] Note - The st(v) syntax descriptor is used in this disclosure only when the current position in the bitstream is a byte-aligned position.

[0120] - tu(v) is a truncated unary encoding scheme that uses up to the number of bits defined by the maximum value (maxVal), and maxVal is defined in the semantics of the corresponding syntax element.

[0121] - u(n) is an unsigned integer using n bits. If n is marked as "v" in the syntax table, the number of bits varies depending on the values ​​of other syntax elements. The parsing process of this descriptor is defined by interpreting the return value of the function read_bits(n) as a binary representation of an unsigned integer where the most significant bit is written first.

[0122] - ue(v) is a 0th-order Exp-Golomb encoded unsigned integer syntax element, written starting from the left bit. The parsing process of this descriptor is defined with order k set to 0.

[0123] Below, SEI messages related to the present invention will be described.

[0124] Table 1 shows an example of a Quality Metric SEI message syntax according to one embodiment.

[0125] [Table 1]

[0126]

[0127] An example of the semantics of a quality indicator SEI message according to one embodiment is described.

[0128] The quality metrics SEI message signals quality metric values ​​representing any one of the following.

[0129] - Single picture quality.

[0130] - Average quality of all pictures corresponding to CLVS.

[0131] Quality gain of a single picture. This is the difference in quality of a single picture relative to the quality of a gain reference picture.

[0132] - Average quality gain of all pictures corresponding to CLVS.

[0133] The use of this SEI message requires the definition of the following variables:

[0134] - Chroma format indicator denoted as ChromaFormatIdc.

[0135] - Number of pictures NumPics.

[0136] - A list of picture widths and heights in units of luma samples. Here, denoted as PicWidth[i] and PicHeight[i] respectively, where i is in the range from 0 to NumPics - 1 (inclusive).

[0137] - A list of pictures TestPicList[i]. Here, i is in the range from 0 to NumPics - 1 (inclusive).

[0138] - When any qm_gain_flag[i] is equal to 1, GainRefPicList[i] is a list of gain-referenced pictures for i in the range from 0 to NumPics - 1 (inclusive).

[0139] The variables SubWidthC and SubHeightC are derived from ChromaFormatIdc.

[0140] TestPicList[i][cIdx] and ReferencePicList[i][cIdx] represent the cIdx-th sample array of the i-th picture in TestPicList and ReferencePicList, respectively.

[0141] TestPicList[i][cIdx][x][y] and ReferencePicList[i][cIdx][x][y] represent the sample at position (x, y) within the cIdx-th sample array of the i-th picture in TestPicList and ReferencePicList, respectively. Here, x is in the range from 0 to ( ( cIdx = = 0 ) ? PicWidth[i] : PicWidth[i] / SubWidthC ) - 1 (inclusive), and y is in the range from 0 to ( ( cIdx = = 0 ) ? PicHeight[i] : PicHeight[i] / SubHeightC ) - 1 (inclusive).

[0142] currPicIdx is set to a value where the output time of TestPicList[currPicIdx] is equal to the output time of the current picture.

[0143] qm_metric_definitions_present_flag, which is equivalent to 1, indicates that information defining quality metrics exists. qm_metric_definitions_present_flag, which is equivalent to 0, indicates that information defining quality metrics does not exist.

[0144] When this SEI message is the first quality metric SEI message in CLVS in decoding order, qm_metric_definitions_present_flag must be equal to 1.

[0145] Otherwise (if this SEI message is not the first quality indicator SEI message in CLVS in the decoding order), at least one of the following two conditions must be satisfied, which is a requirement for bitstream conformance.

[0146] - qm_metric_definitions_present_flag will be equal to 0

[0147] - If the values ​​of the syntax elements qm_metric_type[i], qm_three_component_flag[i], qm_gain_flag[i], qm_gain_reference_flag[i], qm_metric_increasing_flag[i], qm_full_reference_flag[i], qm_value_len_minus1_in_bytes[i], qm_metric_description_present_flag[i], and qm_metric_description[i] exist, they must be equal to the respective syntax elements of the first quality metric SEI message in CLVS.

[0148] A qm_clvs_values_present_flag equal to 1 indicates that qm_clvs_metric_value[i][c] syntax elements exist. A qm_clvs_values_present_flag equal to 0 indicates that qm_clvs_metric_value[i][c] syntax elements do not exist.

[0149] A qm_pic_values_present_flag equal to 1 indicates that qm_pic_metric_value[i][c] syntax elements exist. A qm_pic_values_present_flag equal to 0 indicates that qm_pic_metric_value[i][c] syntax elements do not exist.

[0150] The value obtained by adding 1 to qm_num_metrics_minus1 specifies the number of signaling quality metric items.

[0151] A qm_gain_enabled_flag equal to 1 indicates that the qm_gain_flag[i] syntax element exists. A qm_gain_enabled_flag equal to 0 indicates that the qm_gain_flag[i] syntax element does not exist.

[0152] If qm_gain_flag[i] and qm_gain_reference_flag[i] exist, they represent the interpretation of the values ​​of the syntax elements qm_clvs_metric_value[i][c] and qm_pic_metric_value[i][c]. If qm_gain_flag[i] and qm_gain_reference_flag[i] do not exist, their values ​​are inferred to be equal to 0.

[0153] qm_metric_type[i] specifies the quality metric type of the i-th item as specified in Table 2 below. The value of qm_metric_type[i] must be in the range of 0 to 8 (inclusive) or 128 to 255 (inclusive) within the bitstream suitable for the present disclosure. Values ​​in the range of 9 to 127 (inclusive) for qm_metric_type[i] are reserved for future use and must not exist in the bitstream suitable for the present disclosure.

[0154] If the value of qm_metric_type[i] is in the range from 9 to 127, a decoder suitable for the present disclosure must ignore all syntax elements for the i-th item in this syntax structure.

[0155] If the value of qm_metric_type[i] is in the range of 128 to 255, the quality metric type is unspecified or specified by other means not specified in this disclosure.

[0156] Table 2 below defines the interpretation for qm_metric_type[i].

[0157] [Table 2]

[0158]

[0159] qm_three_component_flag[i] equal to 1 indicates that there will be three component values ​​for the i-th indicator. qm_three_component_flag[i] equal to 0 indicates that there will be a single value for the i-th indicator. It is a requirement of bitstream conformance that qm_three_component_flag[i] must be equal to 0 when ChromaFormatIdc is equal to 0.

[0160] qm_metric_increasing_flag[i] equal to 1 means that a higher value of the i-th metric indicates an improvement in quality. qm_metric_increasing_flag[i] equal to 0 means that a lower value of the i-th metric indicates an improvement in quality. When not present, the value of qm_metric_increasing_flag[i] is inferred to be equal to IncreasingFlag[qm_metric_type[i]] in Table 2.

[0161] qm_full_reference_flag[i], which is equal to 1, indicates that the quality metric is a full reference quality metric calculated by comparing the pictures in TestPicList with their respective quality reference pictures. qm_full_reference_flag[i], which is equal to 0, indicates that the quality metric may or may not include comparing the pictures in TestPicList with their respective quality reference pictures. When not present, the value of qm_full_reference_flag[i] is inferred to be equal to FullReferenceFlag[qm_metric_type[i]] in Table 2.

[0162] The value obtained by adding 1 to qm_value_len_minus1_in_bytes[i] represents the length in bytes of the syntax element qm_pic_metric_value[i][c]. If the value does not exist, the value of qm_value_len_minus1_in_bytes[i] is inferred to be equal to NumBytes[qm_metric_type[i]] - 1.

[0163] qm_metric_description_present_flag[i] equal to 1 indicates that qm_metric_description[i] exists. qm_metric_description_present_flag[i] equal to 0 indicates that qm_metric_description[i] does not exist.

[0164] qm_bit_equal_to_zero must be equal to 0.

[0165] qm_metric_description[i] represents the text description for the i-th quality metric. The length of the syntax element must be less than or equal to 4097 bytes, excluding the null-terminated byte.

[0166] qm_clvs_metric_value[i][c] specifies the mean value of the i-th quality metric for the c-th component of CLVS. The length of the corresponding syntax element is 8 * (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0167] qm_pic_metric_value[i][c] specifies the value of the i-th quality metric for the c-th component of the current picture. The length of the corresponding syntax element is 8 * (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0168] The meaning of the quality metric value is determined by the values ​​of qm_metric_type[i], qm_gain_flag[i], and qm_gain_reference_flag[i].

[0169] When qm_pic_values_present_flag is equal to 1, qm_pic_metric_value[i][c] represents the picture metric value picMetricValue[i][c] of type qm_metric_type[i] described in Table 2 above, and the following applies.

[0170] - When qm_gain_flag[i] is equal to 0, qm_pic_metric_value[i][c] has a value derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0171] - Otherwise (if qm_gain_flag[i] is equal to 1), the following applies.

[0172] - picMetricValueTest[i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0173] - picMetricValueGainRef[i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0174] - qm_pic_metric_value[i][c] has the value of picMetricValueTest[i][c] - picMetricValueGainRef[i][c].

[0175] When qm_clvs_values_present_flag is equal to 1, qm_clvs_metric_value[i][c] represents listPicMetricValue[j][i][c], which is the mean value of picture metric values ​​calculated for all pictures in TestPicList of type qm_metric_type[i] as described in Table 2 above. Here, each listPicMetricValue[j][i][c] is derived as follows for each value of j in the range from 0 to NumPics - 1 (inclusive).

[0176] - When qm_gain_flag[i] is equal to 0, listPicMetricValue[j][i][c] is equal to picMetricValue[i][c] derived according to the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[j], PicWidth[j], and PicHeight[j], respectively.

[0177] - Otherwise (qm_gain_flag[i] is equal to 1), the following applies:

[0178] - listPicMetricValueTest[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0179] - picMetricValueGainRef[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0180] - qm_pic_metric_value[j][i][c] has the value of picMetricValueTest[j][i][c] - picMetricValueGainRef[j][i][c].

[0181] The derivation process of picMetricValue[i][c] is explained.

[0182] The inputs for this process are the tested picture testPic, the picture width picWidth in luma samples, and the picture height picHeight in luma samples.

[0183] The quality reference picture, referencePic, is given as input to the encoding system and is said to be the original picture having the same output time as testPic.

[0184] testPic[cIdx] and referencePic[cIdx] represent the cIdx-th sample arrays of testPic and referencePic, respectively.

[0185] testPic[cIdx][x][y] and referencePic[cIdx][x][y] represent samples at position (x, y) within the cIdx-th sample array of testPic and referencePic, respectively.

[0186] The picture quality metric picMetricValue[i][c] is derived as follows.

[0187] - When qm_metric_type[i] is equal to 0,

[0188] - listPicMetricValueTest[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0189] - picMetricValueGainRef[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0190] - When qm_metric_increasing_flag[i] is equal to 1, a higher value of picMetricValue[i][c] indicates that testPic is of better quality than a picture with a lower picMetricValue[i][c] value.

[0191] - When qm_full_reference_flag[i] is equal to 1, picMetricValue[i][c] represents the quality metric value calculated from the comparison between testPic and referencePic.

[0192] - qm_metric_description[i] provides a text description of the quality metric displayed by picMetricValue[i][c].

[0193] - Further interpretation of picMetricValue[i][c] is determined by external means not specified in this specification.

[0194] - When qm_metric_type[i] is equal to 1,

[0195] - picMetricValue[i][0] is set to the same as the PSNR value calculated using Section 9.4.2 and Section D.2 of ISO / IEC 23001-11 [2] for the lumina components of testPic and referencePic with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0196] - When qm_three_component_flag[i] is equal to 1, - picMetricValue[i][1] and picMetricValue[i][2] are set to have the bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are equal to the PSNR values ​​calculated for the Cb and Cr components of testPic and referencePic, respectively, using sections 9.4.2 and D.2 of ISO / IEC 23001-11 [2].

[0197] - When qm_metric_type[i] is equal to 2,

[0198] - picMetricValue[i][0] is set to be equal to the value of the variable psnrYUV calculated as follows.

[0199] - The variable psnrY is set to the same as the PSNR value calculated using Section 9.4.2 and Section D.2 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0200] - Variables psnrU and psnrV are set to have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are equal to the PSNR values ​​calculated using sections 9.4.2 and D.2 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0201] [Formula 1]

[0202]

[0203] - When qm_metric_type[i] is equal to 3,

[0204] - picMetricValue[i][0] is set to the same as the SSIM value calculated from Section 9.4.2 and Section D.5 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic, with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0205] - When qm_three_component_flag[i] is equal to 1,

[0206] - picMetricValue[i][1] and picMetricValue[i][2] have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are set to be equal to the SSIM values ​​calculated from Section 9.4.2 and Section D.5 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0207] - When qm_metric_type[i] is equal to 4,

[0208] - picMetricValue[i][0] is set to the same as the MS-SSIM value calculated from Section 4.3.3 of ISO / IEC 23001-10 [1] for the luminance components of testPic and referencePic with the bit depth OrigBitDepth.

[0209] - When qm_three_component_flag[i] is equal to 1,

[0210] - picMetricValue[i][1] and picMetricValue[i][2] have a bit depth of OrigBitDepth and are set to be equal to the MS-SSIM values ​​calculated from Section 4.3.3 of ISO / IEC 23001-10 [1] for the Cb and Cr components of testPic and referencePic, respectively.

[0211] - When qm_metric_type[i] is equal to 5,

[0212] - picMetricValue[i][0] is set to the same as the MOS value specified in Section 4.3.6 of ISO / IEC 23001-10 [1].

[0213] - When qm_metric_type[i] is equal to 6,

[0214] - picMetricValue[i][0] is set to the same as the wPSNR value calculated from Section 9.4.2 and Section D.3 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic, with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0215] - When qm_three_component_flag[i] is equal to 1,

[0216] - picMetricValue[i][1] and picMetricValue[i][2] have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are set to be equal to the wPSNR values ​​calculated from Section 9.4.2 and Section D.3 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0217] - When qm_metric_type[i] is equal to 7,

[0218] - picMetricValue[i][0] is set to the same WS-PSNR value calculated from Section 9.4.2 and Section D.4 of ISO / IEC 23001-11 [2] for the luminance components of testPic and referencePic with bit depth OrigBitDepth, width picWidth, and height picHeight.

[0219] - When qm_three_component_flag[i] is equal to 1,

[0220] - picMetricValue[i][1] and picMetricValue[i][2] have bit depth OrigBitDepth, width picWidth / SubWidthC, and height picHeight / SubHeightC, and are set to be equal to the WS-PSNR values ​​calculated from Section 9.4.2 and Section D.4 of ISO / IEC 23001-11 [2] for the Cb and Cr components of testPic and referencePic, respectively.

[0221] - When qm_metric_type[i] is equal to 8,

[0222] - picMetricValue[i][0] is set to the value of lumaMse derived as follows, which is interpreted as a floating-point value.

[0223] [Equation 2]

[0224]

[0225] If qm_three_component_flag[i] is equal to 1, picMetricValue[i][1] and picMetricValue[i][2] are set to the values ​​of CbMse and CrMse, respectively derived as follows, which are interpreted as floating-point values.

[0226] [Equation 3]

[0227]

[0228] - When qm_metric_type[i] is equal to 9,

[0229] - picMetricValue[i][0] is set to equal the VMAF calculated from Section 4.3.3 of ISO / IEC 23001-10 [1] for the luminance components of testPic and referencePic with the bit depth OrigBitDepth. VMAF is a full reference metric that uses machine learning to fuse scores from multiple elementary quality metrics to generate a quality score for the video. This metric is trained to simulate a quality evaluation obtained as a result of a subjective test. The range of values ​​obtained from the standard (HD) VMAF model is [0,..100].

[0230] - When qm_metric_description_present_flag[i] is equal to 1, qm_metric_description[i] may contain information about the version of the VMAF metric, such as the metric release number and the VMAF model.

[0231] The use of the quality metric SEI message is explained.

[0232] For the purpose of interpreting quality metric SEI messages, the derivation of the variables ChromaFormatIdc, NumPics, TestPicList, PicWidth, PicHeight, and GainRefPicList is specified as follows.

[0233] TestPicList is initially composed of cropped decoded pictures of the current CLVS in output order.

[0234] When a quality indicator SEI message is included as an SEI message type within an SEI processing order SEI message, any quality indicator SEI message associated with that SEI processing order SEI message must be included within a processing order nesting SEI message.

[0235] When a quality indicator SEI message is included as the i-th SEI message type within an SEI processing sequence SEI message, the following applies.

[0236] - It is a bitstream conformance requirement that the SEI message seiB, which implies post-processing to be performed, must exist as the j-th SEI message type within the same SEI processing order SEI message, and that po_sei_processing_order[j] must be equal to po_sei_processing_order[i].

[0237] - The corresponding quality indicator SEI message represents the picture quality resulting from the post-processing implied by seiB.

[0238] - TestPicList is updated as follows for each post-processing step where po_sei_processing_order[j] is less than or equal to po_sei_processing_order[i], in the non-decreasing order of j.

[0239] - When the post-processing step generates a picture picA as a result that has the same output time as any picture picB in TestPicList, picB in TestPicList is replaced with picA.

[0240] - When the post-processing step generates a picture picA as a result that has an output time not equal to the output time of any picture in TestPicList, picA is inserted into TestPicList in such a way that the pictures in TestPicList maintain their output order.

[0241] NumPics is set to be equal to the number of pictures in TestPicList.

[0242] PicWidth[i] and PicHeight[i] are each set to be equal to the width and height of TestPicList[i] in units of luma samples.

[0243] When a quality metric SEI message is included as the k-th SEI message within a processing order nesting SEI message and qm_gain_flag[i] is equal to 1 for any value of i, the following applies.

[0244] - When qm_gain_reference_flag[i] is equal to 0, the i-th metric value in the quality metric SEI message represents the gain of the post-processing step where po_sei_processing_order[j] is equal to pon_processing_order[k] for the picture or pictures used as input to the corresponding post-processing step, and GainRefPicList is set to be equal to TestPicList derived for the processing steps up to that point, excluding the processing step with po_sei_processing_order[j].

[0245] - Otherwise (when qm_gain_reference_flag[i] is equal to 1), the i-th metric value in the quality metric SEI message represents the cumulative gain of all post-processing steps for the clipped decoded picture or pictures for which po_sei_processing_order[j] is less than or equal to pon_processing_order[k], and GainRefPicList consists of the clipped decoded pictures of the current CLVS in output order.

[0246] - Bitstream conformance requirements state that the number of pictures in GainRefPicList must be equal to NumPics, and that the width and height of GainRefPicList[i] in luma samples must be equal to PicWidth[i] and PicHeight[i], respectively.

[0247] The value of ChromaFormatIdc is derived as follows.

[0248] - If a quality metric SEI message is not included in a PON (Processing Order Nesting) SEI message, ChromaFormatIdc is set to be equal to sps_chroma_format_idc.

[0249] - Otherwise (when the quality indicator SEI message is included in the PON SEI message), ChromaFormatIdc is set to a value that matches the chroma format of the pictures in TestPicList, and it is a bitstream conformance requirement that the chroma formats of all pictures in TestPicList and GainRefPicList (if present) must be identical.

[0250] When the quality metric SEI message qmSeiA exists in a picture unit other than the first picture unit of CLVS in the decoding order and at least one value of qm_clvs_metric_value[i][c] exists, the following applies.

[0251] - If qmSeiA is not included in the PON SEI message, each value of qm_clvs_metric_value[i][c] must be equal to the value of qm_clvs_metric_value[i][c] in the quality metric SEI message that exists in the first picture unit of CLVS and is not included in the processing order nesting SEI message.

[0252] - Otherwise (where qmSeiA is included within a PON SEI message containing a specific set of pon_target_po_id[i] values), the following applies.

[0253] - It is a requirement of bitstream conformance that there must be a quality indicator SEI message qmSeiB contained within a processing order nesting SEI message that exists in the first picture unit of CLVS and has the same set of pon_target_po_id[i] values.

[0254] - Each value of qm_clvs_metric_value[i][c] in qmSeiA must be equal to the value of qm_clvs_metric_value[i][c] in qmSeiB.

[0255] Conventional Quality Metrics (QM) SEI messages involve signaling the average quality of CLVS through the signaling of the syntax elements qm_clvs_values_present_flag and qm_clvs_metric_value[i][c]. The semantics of these syntax elements are as follows.

[0256] ...

[0257] If qm_clvs_values_present_flag is 1, it indicates that the qm_clvs_metric_value[i][c] syntax element exists. If qm_clvs_values_present_flag is 0, it indicates that the qm_clvs_metric_value[i][c] syntax element does not exist.

[0258] qm_clvs_metric_value[i][c] represents the average value of the i-th quality metric for the c-th component of CLVS. The length of the corresponding syntax element is 8 × (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0259] ...

[0260] If the quality metric SEI message qmSeiA exists in a picture unit other than the first picture unit of CLVS in the decoding order, and there is one or more values ​​of qm_clvs_metric_value[i][c], the following applies.

[0261] - If qmSeiA is not included in the PON (processing order nesting) SEI message, each qm_clvs_metric_value[i][c] value must be the same as the qm_clvs_metric_value[i][c] value of the quality metric SEI message that exists in the first picture unit of CLVS and is not included in the PON SEI message.

[0262] - Otherwise (if qmSeiA is included in a PON SEI message with a specific set of pon_target_po_id[i] values), the following applies.

[0263] - As a bitstream conformance requirement, there must be a quality indicator SEI message qmSeiB that is included in a PON SEI message with the same set of pon_target_po_id[i] values ​​and exists in the first picture unit of CLVS.

[0264] - Each qm_clvs_metric_value[i][c] value of qmSeiA must be the same as the qm_clvs_metric_value[i][c] value of qmSeiB.

[0265] ...

[0266] Given the constraint that the average quality metric value must be the same across the entire CLVS, the problem is that if one or more pictures are removed from the bitstream (e.g., when performing temporal sub-layer-based bitstream extraction), all QM SEI messages within the CLVS must be updated.

[0267] It would be desirable to eliminate the need for such QM SEI message updates.

[0268] In one embodiment, the following items may be applied individually or in combination.

[0269] Modify the signaling of average quality metric values ​​to be based on temporal IDs rather than a single value for all pictures in CLVS.

[0270] One embodiment relates to the above-described item. One embodiment is based on the VSEI standard.

[0271] An example of the semantics of a quality indicator SEI message according to one embodiment is described. Semantics different from the semantics of the quality indicator SEI message described above are described, and the description of semantics identical to the semantics of the quality indicator SEI message described above is replaced by the description of the semantics of the quality indicator SEI message described above.

[0272] ...

[0273] The syntax elements of the Quality Metric SEI message according to one embodiment are as shown in Table 3 below.

[0274] [Table 3]

[0275]

[0276] ...

[0277] The value obtained by adding 1 to qm_max_sublayers_minus1 represents the maximum number of temporal sub-layers for signaling the average quality metric of CLVS.

[0278] qm_num_pic_in_tid[k] represents the number of pictures in TestPicList with a temporal ID less than or equal to k, if present. The value of qm_num_pic_in_tid[k] ranges from 0 to 2 32 - Must be within the range of 2 (inclusive).

[0279] ...

[0280] qm_clvs_metric_value[k][i][c] specifies the average value of the i-th quality metric for the c-th component for pictures in CLVS with a temporal ID of k or less. The length of the corresponding syntax element is 8 × (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0281] ...

[0282] When qm_clvs_values_present_flag is 1, qm_clvs_metric_value[k][i][c] represents listPicMetricValue[j][i][c], the mean value of picture metric values ​​calculated for all pictures in TestPicList with a temporal ID of k or less, of type qm_metric_type[i] as described in Table 2. Here, each listPicMetricValue[j][i][c] is derived as follows for each value of j in the range from 0 to NumPics - 1 (inclusive).

[0283] - When qm_gain_flag[i] is 0, listPicMetricValue[j][i][c] is equal to picMetrictValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight allocated to TestPicList[j], PicWidth[j], and PicHeight[j], respectively.

[0284] - Otherwise (when qm_gain_flag[i] is 1), the following applies.

[0285] - listPicMetricValueTest[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0286] - picMetricValueGainRef[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0287] - qm_pic_metric_value[j][i][c] has the value of picMetricValueTest[j][i][c] - picMetricValueGainRef[j][i][c].

[0288] ...

[0289] One embodiment relates to the above-described item. One embodiment is based on the VSEI standard.

[0290] An example of the semantics of a quality indicator SEI message according to one embodiment is described. Semantics different from the semantics of the quality indicator SEI message described above are described, and the description of semantics identical to the semantics of the quality indicator SEI message described above is replaced by the description of the semantics of the quality indicator SEI message described above.

[0291] ...

[0292] The syntax elements of a Quality Metric SEI message according to one embodiment are as shown in Table 4 below.

[0293] [Table 4]

[0294]

[0295] ...

[0296] The value obtained by adding 1 to qm_max_sublayers_minus1 represents the maximum number of temporal sub-layers for signaling the average quality metric of CLVS.

[0297] qm_num_pic_in_tid[k] represents the number of pictures in TestPicList with a temporal ID less than or equal to k, if present. The value of qm_num_pic_in_tid[k] ranges from 0 to 2 32 - Must be within the range of 2 (inclusive).

[0298] ...

[0299] qm_clvs_metric_value[k][i][c] specifies the average value of the i-th quality metric for the c-th component for pictures in CLVS with a temporal ID of k or less. The length of the corresponding syntax element is 8 × (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0300] ...

[0301] When qm_clvs_values_present_flag is 1, qm_clvs_metric_value[k][i][c] represents listPicMetricValue[j][i][c], the mean value of picture metric values ​​calculated for all pictures in TestPicList with a temporal ID of k or less, of type qm_metric_type[i] as described in Table 2. Here, each listPicMetricValue[j][i][c] is derived as follows for each value of j in the range from 0 to NumPics - 1 (inclusive).

[0302] - When qm_gain_flag[i] is 0, listPicMetricValue[j][i][c] is equal to picMetrictValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight allocated to TestPicList[j], PicWidth[j], and PicHeight[j], respectively.

[0303] - Otherwise (when qm_gain_flag[i] is 1), the following applies.

[0304] - listPicMetricValueTest[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to TestPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0305] - picMetricValueGainRef[j][i][c] is set to be equal to picMetricValue[i][c] derived by the process specified in this disclosure, with testPic, picWidth, and picHeight assigned to become GainRefPicList[currPicIdx], PicWidth[currPicIdx], and PicHeight[currPicIdx], respectively.

[0306] - qm_pic_metric_value[j][i][c] has the value of picMetricValueTest[j][i][c] - picMetricValueGainRef[j][i][c].

[0307] ...

[0308] The terms or names described below (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms, etc. described below. For example, the image information described below may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0309] The operations described below do not constitute an essential component of one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of one embodiment, and previously described operations may be added. Moreover, unless they contradict previously described operations, the operations described below form one embodiment integrally with previously described operations and do not form a separate embodiment distinct from previously described operations.

[0310] FIG. 5 is a diagram illustrating a method for decoding image information according to one embodiment of the present disclosure.

[0311] The decoding method (S500) may include the operations described below.

[0312] The terms or names described below (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms or names described below. For example, the image information described below may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0313] The operations described below do not constitute an essential component of the decoding method according to one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of the decoding method according to one embodiment, and the previously described operations may be added.

[0314] The operations described below form a single embodiment integrally with the configurations and / or operations described above, unless they conflict with the configurations and / or operations described above, and do not form a separate embodiment distinct from the configurations and / or operations described above.

[0315] The decoding method (S500) can be executed by a decoding device including a memory and a processor electrically connected to the memory, for example, by a processor.

[0316] A decoding device can acquire image information. For example, a processor of a decoding device can acquire image information. The image information may include a picture to be decoded.

[0317] For example, the image information may include additional information related to at least one picture (e.g., analysis information that can be used to calculate quality indicators). Additionally, quality indicator information calculated by an encoding device may be provided for quality evaluation of at least one picture. The encoding device may perform image analysis on the picture and generate statistical values ​​or quality evaluation results necessary for calculating quality indicator information, and provide them to a decoding device. Accordingly, since the decoding device utilizes the quality indicators provided by the encoding device without the need to perform separate quality analysis operations, the amount of computation and power consumption during the decoding process may be reduced.

[0318] The decoding device can acquire a SEI (supplemental enhancement information) message (S510).

[0319] SEI messages can convey specific types of information that assist in processes related to the decoding, display, or other purposes of image information. Here, SEI messages may not be necessary for the decoding process to determine the sample values ​​of the decoded picture.

[0320] For example, a decoding device may acquire at least one SEI message associated with a picture from a bitstream. For example, a processor of the decoding device may acquire at least one SEI message associated with a picture. Here, the picture represents an image frame or field encoded within the bitstream and may include sample arrays consisting of a luminance component and one or more chroma components. Additionally, the picture has a decoding order and an output order, and the decoding device may reconstruct the picture using syntax elements within the bitstream.

[0321] Here, the SEI message may include a QM SEI message that provides quality metric information included in at least one picture unit. Specifically, the SEI message may be a QM SEI message that provides at least one (or multiple) quality metric information included in at least one picture unit. Specifically, the QM SEI message may include at least one (or multiple) quality metric information included in at least one picture unit.

[0322] For example, the decoding device can obtain a QM SEI message contained in at least one picture unit from the bitstream.

[0323] The QM SEI message may include at least one of information regarding the quality of a single picture, information regarding the average quality of all pictures corresponding to the CLVS, information regarding the quality gain of a single picture, and / or information regarding the average quality gain of all pictures corresponding to the CLVS. The information regarding quality may be expressed in various ways, such as quality information, quality indicator information, or quality indicators, and may include subjective or objective indicators regarding the quality of the picture.

[0324] QM SEI messages may have various names, such as quality indicator SEI messages, quality indicator related messages, quality indicator related information, quality indicator messages, QM messages, QM related messages, and QM related information, and such names are not limited.

[0325] QM SEI messages can take various forms. For example, a QM SEI message may be a syntax element or a syntax structure containing one or more syntax elements. Additionally, a QM SEI message may be a raw byte sequence payload (RBSP) containing one or more syntax elements or one or more syntax structures. For example, a QM SEI message may be represented as quality_metric(payloadSize), but is not limited thereto.

[0326] The decoding device can obtain quality indicator information based on the SEI message (S520).

[0327] For example, the decoding device can obtain quality indicator information for the decoded picture based on at least one QM SEI message.

[0328] For example, the quality indicator information provided by the QM SEI message may include information on whether a definition of the quality indicator exists, information regarding at least one sub-layer for the picture unit, information on whether average quality exists, average quality indicator information, information on the number of quality indicator entries, quality indicator component information, quality indicator type information, and information on whether there is a quality gain.

[0329] For example, the processor of the decoding device can process a QM SEI message. The processor of the decoding device can obtain quality indicator information for a picture based on information regarding the existence of a definition for a quality indicator included in the QM SEI message, information regarding at least one sub-layer for a picture unit, information regarding the existence of average quality, average quality indicator information, information on the number of quality indicator entries, quality indicator type information, and information regarding quality gain.

[0330] The decoding device can obtain information regarding at least one sublayer for the picture unit from the QM SEI message.

[0331] For example, information regarding at least one sublayer for a picture unit can be obtained based on whether information regarding the existence of average quality indicates that information regarding the average quality indicator exists. That is, if information regarding the average quality indicator exists within the QM SEI message, the decoding device can obtain information regarding at least one sublayer for a picture unit.

[0332] For example, information regarding at least one sub-layer for a picture unit may include information on the maximum number of at least one sub-layer and information on the number of pictures included in at least one sub-layer.

[0333] Information on the maximum number of sub-layers can represent the maximum number of temporal sub-layers used to signal average quality metric information for CLVS in units of temporal sub-layers.

[0334] Information regarding the maximum number of sublayers may take various forms and be expressed by various names. For example, information regarding the maximum number of sublayers may be a syntax element or a syntax structure containing one or more syntax elements. For example, information regarding the maximum number of sublayers may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, information regarding the maximum number of sublayers may include qm_max_sublayers_minus1, but is not limited thereto.

[0335] For example, if the information on the maximum number of sublayers is expressed as qm_max_sublayers_minus1, the value obtained by adding 1 to qm_max_sublayers_minus1 may represent the maximum number of temporal sublayers for signaling average quality metric information within CLVS. For example, if the value of qm_max_sublayer_minus1 is 2, the sublayers will include a total of 3 layers of 0, 1, and 2, and for each layer of the sublayer, the value of the average quality metric information can be derived for all pictures having a temporal ID that is less than or equal to the layer.

[0336] The picture count information included in the sub-layer can represent the number of valid pictures corresponding to each temporal sub-layer in order to calculate the value of the average quality indicator information at the temporal sub-layer level within CLVS.

[0337] For example, information on the number of pictures included in at least one sublayer may represent the number of at least one picture having an identifier that is less than or equal to the identifier corresponding to that sublayer. That is, information on the number of pictures included in a temporal sublayer may represent the number of all pictures having a temporal identifier that is equal to or smaller than the temporal identifier (temporal ID) of that sublayer. In other words, information on the number of pictures included in a sublayer may represent the number of pictures referenced to calculate the average quality metric for each sublayer.

[0338] The picture count information included in the sublayer may take various forms and may be expressed by various names. For example, the picture count information included in the sublayer may be a syntax element or a syntax structure containing one or more syntax elements. For example, the picture count information included in the sublayer may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, the picture count information included in the sublayer may include qm_num_pic_in_tid[k], but is not limited thereto.

[0339] For example, if the information regarding the number of pictures included in a sublayer is represented by qm_num_pic_in_tid[k], then if such picture count information exists, qm_num_pic_in_tid[k] may represent the number of pictures included in TestPicList for which the temporal sublayer identifier is k or less. Here, k may represent the temporal ID of the temporal sublayer. For example, the value of qm_num_pic_in_tid[k] ranges from 0 to 2 32 - It can be in the range of up to 2 (inclusive).

[0340] For example, information on the number of pictures included in at least one sublayer can be obtained based on the number of sublayers according to the information on the maximum number of sublayers.

[0341] For example, information on the number of pictures included in at least one sublayer can be repeatedly obtained as many times as the number of sublayers determined by the information on the maximum number of sublayers obtained from the QM SEI message. That is, based on the number of sublayers determined by the information on the maximum number of sublayers obtained from the QM SEI message, the number of pictures having a temporal ID below that layer can be sequentially obtained for each sublayer layer. Through this, the decoding device can obtain information on the number of pictures in each sublayer, indicating the number of accumulated pictures included up to that layer. The decoding device obtains information on the number of pictures included in each sublayer based on the accumulated number of pictures and can calculate an average quality indicator value at the sublayer level using this information.

[0342] For example, if the maximum number information of the sublayer indicates that there are three temporal sublayers, the decoding device can sequentially obtain picture count information for layers 0, 1, and 2 of the sublayer, respectively. At this time, the picture count information corresponding to layer 0 may represent the number of pictures with a temporal ID of 0, the picture count information corresponding to layer 1 may represent the cumulative number of pictures with a temporal ID of 0 or 1, and the picture count information corresponding to layer 2 may represent the cumulative number of pictures with a temporal ID of 0, 1, or 2.

[0343] The decoding device can obtain average quality indicator information corresponding to at least one sublayer for the picture unit from the QM SEI message.

[0344] Average quality metric information may represent the average quality metric for the pictures included in CLVS. For example, the average quality metric information may represent the absolute average quality based on the value of the quality gain information, or the relative average quality relative to the reference. For example, the average quality metric information may be a user-defined quality metric based on the value of the quality metric type information, or PSNR, PSNR-YUV, SSIM, MS-SSIM, MOS, wPSNR, WS-PSNR, MSE, or VMAF.

[0345] Here, the quality indicator type information may indicate what type of measurement method the picture quality indicator information and / or average quality indicator information represents. For example, the quality indicator type information may indicate whether the picture quality indicator information and / or average quality indicator information is a user-defined quality indicator, PSNR (Peak Signal-to-Noise Ratio), PSNR-YUV (PSNR integrating Y, U, and V components), SSIM (Structural Similarity), MS-SSIM (Multi-Scale SSIM), MOS (Mean Opinion Score), wPSNR (Weighted PSNR), WS-PSNR (Weighted Spherical PSNR), MSE (Mean Squared Error), or VMAF (Video Multimethod Assessment Fusion). The quality indicator type information is not limited to the types of quality indicators exemplified above and may indicate any unexemplified quality indicator capable of representing the quality of the image.

[0346] Quality metric type information may take various forms and may be expressed by various names. For example, quality metric type information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality metric type information that is a syntax element may include syntax elements such as qm_metric_type[i], but is not limited thereto. For example, when quality metric type information is expressed as qm_metric_type[i], qm_metric_type[i] may represent quality metric type information of the i-th quality metric entry.

[0347] Average quality metric information may take various forms and may be represented by various names. For example, average quality metric information may be a syntax element or a syntax structure containing one or more syntax elements. For example, average quality metric information may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, average quality metric information may include qm_clvs_metric_value[i], but is not limited thereto.

[0348] For example, when average quality metric information is represented as qm_clvs_metric_value[k][i][c], qm_clvs_metric_value[k][i][c] may represent the average value of the i-th quality metric entry for the c-th component, calculated for all pictures in the CLVS where the temporal ID is k or less. Here, index k represents the temporal ID of the temporal sub-layer, index i is an index identifying the type of quality metric entry and can be calculated based on the number of quality metric entries, and index c may be a component index determined based on quality metric component information indicating whether the quality metric is composed of a single component or three color components (Y, U, V, etc.).

[0349] Here, the information regarding the number of quality metric entries may represent the number of quality metric entries generated within the QM SEI message. Information regarding the number of quality metric entries may take various forms and may be expressed by various names. For example, information regarding the number of quality metric entries may be a syntax element or a syntax structure containing one or more syntax elements. For example, information regarding the number of quality metric entries that is a syntax element may be expressed as the syntax element qm_num_metrics_minus1, but is not limited thereto. For instance, if information regarding the number of quality metric entries is expressed as qm_num_metrics_minus1, the value obtained by adding 1 to qm_num_metrics_minus1 may represent the number of signaled quality metric entries.

[0350] Here, quality indicator component information may indicate whether the quality indicator information consists of a single component or three components. The quality indicator component information may take various forms and may be expressed by various names. For example, the quality indicator component information may be a syntax element or a syntax structure containing one or more syntax elements. For example, the quality indicator component information that is a syntax element may be expressed as the syntax element qm_three_component_flag, but is not limited thereto.

[0351] For example, when quality indicator component information is represented by qm_three_component_flag, if the value of qm_three_component_flag is 1, it indicates that the quality indicator information consists of three components. For example, the quality indicator value may include quality indicator values ​​for each of the image's multi-color components, such as Y, U, V components or R, G, B components. In this case, the component index can have three values. On the other hand, if the value of qm_three_component_flag is 0, the quality indicator consists of a single component, and the component index can have one value. Additionally, the length of the syntax element of the average quality indicator information can be 8 * (qm_value_len_minus1_in_bytes[i] + 1) bits.

[0352] For example, average quality indicator information can be derived by averaging the quality indicator values ​​of at least one picture having an identifier that is less than or equal to the identifier corresponding to that sublayer, based on the number of pictures included in at least one sublayer.

[0353] Each sub-layer is determined according to the information on the maximum number of sub-layers signaled within the QM SEI message, and in each layer, pictures having identifiers less than or equal to the identifier of that layer may be determined. For each of these pictures, a quality indicator calculation procedure is performed according to the processor specified in this disclosure, and a quality indicator value for each individual picture may be derived. Subsequently, the average quality indicator value corresponding to each sub-layer can be obtained by calculating an average value by accumulating and summing the quality indicator values ​​calculated for each of the valid pictures based on the valid pictures determined according to the information on the number of pictures included in the corresponding sub-layer, and then dividing the result by the number of valid pictures.

[0354] If average quality indicator information exists within the QM SEI message, the average quality indicator value may represent the average value of the quality indicator values ​​of each picture calculated for pictures in TestPicList whose temporal ID is less than or equal to a specific value, corresponding to the quality indicator types described in Table 2. In this case, the quality indicator value of each picture may be obtained by applying the quality indicator calculation process specified in this specification to individual pictures in TestPicList.

[0355] For example, if the quality metric value of each picture is represented as listPicMetricValue[j][i][c], listPicMetricValue[j][i][c] may represent the i-th quality metric value of the c-th component calculated for the j-th picture in TestPicList. Here, index j represents an index for identifying individual pictures within TestPicList, index i represents an identifier for distinguishing the type of quality metric entry, and index c represents the component to which the quality metric of the corresponding picture is applied.

[0356] Therefore, when the average quality metric information is represented as qm_clvs_metric_value[k][i][c], the average quality metric information can be derived by accumulating the quality metric values ​​of each picture, listPicMetricValue[j][i][c], which are calculated for all pictures that satisfy the condition that the temporal ID is less than or equal to a specific value k, and performing an arithmetic mean.

[0357] In addition, since the sub-layer structure has an accumulative hierarchical characteristic, the pictures considered at sub-layer layer k can include all pictures included in sub-layers 0 through k-1. Accordingly, as k increases, the number of pictures used to calculate the value of the average quality indicator information increases, and consequently, the higher the sub-layer, the more pictures whose quality information is reflected can be reflected in the value of the average quality indicator information. The average quality indicator information calculated based on the accumulative sub-layer structure in this way enables the decoding device to more accurately identify the average quality characteristics corresponding to different frame transmission levels (e.g., 30fps, 60fps, 120fps, etc.) in an environment where temporal scalability is applied.

[0358] The quality indicator value of each picture can be set as follows by the process specified in this disclosure, based on information regarding whether there is a quality gain.

[0359] Information on quality gain may indicate whether picture quality indicator information and / or average quality indicator information represents absolute quality or whether picture quality indicator information and / or average quality indicator information represents relative quality relative to a reference object.

[0360] Quality gain information may take various forms and may be represented by various names. For example, quality gain information may be a syntax element or a syntax structure containing one or more syntax elements. For example, quality gain information may include a flag consisting of one bit, or an indicator consisting of two or more bits or a variable-length bit. For example, quality gain information may include qm_gain_flag[i], but is not limited thereto.

[0361] For example, if the value of the quality gain information is 1, the quality gain information may indicate that the picture quality indicator information and / or average quality indicator information represents relative quality compared to the reference object. Additionally, if the value of the quality gain information is 0, the quality gain information may indicate that the picture quality indicator information and / or average quality indicator information represents absolute quality.

[0362] For example, if the value of the quality gain information is 0, the quality indicator value of each picture (listPicMetricValue[j][i][c]) is set to be the same as the quality indicator value of the picture itself (picMetricValue[i][c]), and if the value of the quality gain information is 1, it may be set to the difference between the two values ​​(picMetricValueTest[j][i][c] - picMetricValueGainRef[j][i][c]) obtained by applying the process specified in the present disclosure to the test picture (TestPicList[currPicIdx]) and the reference picture (GainRefPicList[currPicIdx]), respectively. However, this is not limited thereto, and alternatively, specifying that the value of the quality gain information is 1 may be changed to specifying that the value of the quality gain information is 0.

[0363] For example, average quality indicator information can be obtained based on whether the information on the existence of average quality indicates that the average quality indicator information exists. That is, if the average quality indicator information exists within the QM SEI message, the decoding device can obtain the average quality indicator information.

[0364] Information on the presence of average quality may indicate whether a QM SEI message contains information on average quality indicators for the pictures included in CLVS. Information on the presence of average quality may take various forms and may be expressed by various names. For example, information on the presence of average quality may be a syntax element or a syntax structure containing one or more syntax elements. For example, information on the presence of average quality that is a syntax element may be expressed as the syntax element qm_clvs_values_present_flag, but is not limited thereto.

[0365] Information on the existence of average quality may include a flag consisting of one bit, or an indicator consisting of two or more bits or a variable-length bit. For example, if the value of the information on the existence of average quality is 1, it may indicate that a syntax element of the average quality indicator information exists. Additionally, if the value of the information on the existence of average quality is 0, it may indicate that a syntax element of the average quality indicator information does not exist. However, this is not limited thereto, and alternatively, specifying that the value of the information on the existence of average quality is 1 may be interchangeable with specifying that the value of the information on the existence of average quality is 0.

[0366] For example, information on the maximum number of sublayers and the number of pictures included in the sublayers can be obtained based on information regarding the existence of average quality. The information on the maximum number of sublayers and the number of pictures included in the sublayers can be obtained based on the information regarding the existence of average quality indicating that average quality indicator information exists. That is, if average quality indicator information exists within the QM SEI message, the decoding device can obtain information on the maximum number of sublayers and the number of pictures included in the sublayers.

[0367] Meanwhile, the decoding device can obtain information on whether a definition of a quality indicator exists from the QM SEI message.

[0368] Information regarding the existence of definitions for quality metrics may indicate whether definition information for quality metrics exists within a QM SEI message. Information regarding the existence of definitions for quality metrics may take various forms and may be expressed by various names. For example, information regarding the existence of definitions for quality metrics may be a syntax element or a syntax structure containing one or more syntax elements. Information regarding the existence of definitions for quality metrics may include a 1-bit flag or an indicator of 2 or more bits. For example, information regarding the existence of definitions for quality metrics that is a syntax element may be expressed as the syntax element qm_metric_definitions_present_flag, but is not limited thereto.

[0369] For example, if the value of the information on whether a definition of a quality indicator exists is 1, it may indicate that information defining a quality indicator exists within the QM SEI message. Additionally, if the value of the information on whether a definition of a quality indicator exists is 0, it may indicate that information defining a quality indicator does not exist within the QM SEI message. However, this is not limited thereto, and alternatively, specifying that the value of the information on whether a definition of a quality indicator exists is 1 may be interchangeable with specifying that the value of the information on whether a definition of a quality indicator exists is 0.

[0370] For example, if information regarding the existence of a definition for a quality indicator indicates that definition information for a quality indicator exists within the QM SEI message, and information regarding the existence of an average quality indicates that average quality indicator information exists within the QM SEI message, then information regarding the maximum number of at least one sublayer can be obtained. That is, if definition information for a quality indicator and average quality indicator information exist within the QM SEI message, the decoding device can obtain information regarding the maximum number of at least one sublayer.

[0371] According to the present disclosure, unlike conventional methods that signal a single average quality indicator value for the entire CLVS, by configuring to signal an average quality indicator based on a temporal sub-layer, there is no need to change the value of the average quality indicator even when a picture is removed from the bitstream or only a specific temporal sub-layer is selectively extracted.

[0372] Conventionally, a single average quality metric value for the entire CLVS is used; therefore, when bitstream extraction based on temporal sublayers is performed and some pictures are removed, the average value changes, leading to the problem of having to rewrite all QM SEI messages within the CLVS. In contrast, the present disclosure provides independent average quality metrics for pictures below a temporal ID on a hierarchical basis. Thus, even when only a partial bitstream of a specific layer is used, the decoding device can estimate the average quality metric information using only the average quality metric information of that layer. Consequently, there is no need to recalculate or update QM SEI messages.

[0373] Furthermore, since the calculation of the average quality index value is performed based on a cumulative structure at the sub-layer level, more picture information is reflected in the average calculation as the hierarchy of the sub-layers increases. In other words, because the set of pictures below the temporal identifier corresponding to a specific sub-layer has a cumulative structure that includes pictures contained in all lower sub-layers, the average quality index value calculated corresponding to the upper sub-layer can be derived as a representative value that reflects the quality characteristics of a greater number of pictures.

[0374] FIG. 6 is a diagram illustrating a method for encoding image information according to one embodiment of the present disclosure.

[0375] The terms or names described in FIG. 6 (e.g., names of syntax elements or variables, etc.) are merely examples, and the technical features of the present disclosure are not limited to the terms, etc. described in FIG. 6. For example, the image information described in FIG. 6 may include various information according to the embodiments described in the present disclosure and may include information described in at least one of the tables described above.

[0376] The encoding method (S600) may include operations described below. The operations described below do not constitute an essential component of the encoding method according to one embodiment, and at least some of the operations described below may be omitted. Furthermore, the operations described below do not constitute a sufficient component of the encoding method according to one embodiment, and the previously described operations may be added. Moreover, unless the operations described below contradict the previously described operations, they form an embodiment integrally with the previously described operations and do not form a separate embodiment distinct from the previously described operations.

[0377] The encoding device can generate a SEI (supplemental enhancement information) message (S610).

[0378] For example, the processor of the encoding device can generate a quality indicator for the decoded picture and generate a QM SEI message containing quality indicator information based on the quality indicator for the decoded picture.

[0379] The processor can acquire quality metrics for the decoded picture, including values ​​of user-defined metrics, PSNR (Peak Signal-to-Noise Ratio), PSNR-YUV (PSNR integrating Y, U, and V components), SSIM (Structural Similarity), MS-SSIM (Multi-Scale SSIM), MOS (Mean Opinion Score), wPSNR (Weighted PSNR), WS-PSNR (Weighted Spherical PSNR), MSE (Mean Squared Error), and / or VMAF (Video Multimethod Assessment Fusion). The processor can generate quality metric information (e.g., picture quality metric information and / or average quality metric information) based on the quality metrics.

[0380] The encoding device can perform image analysis on the pictures and generate Quality Metric information to be used for quality evaluation of the pictures. In other words, the encoding device can take a picture as input, calculate a Quality Metric value corresponding to a specific Quality Metric type, and generate this as a QM SEI message. By having the encoding device calculate Quality Metric information in this way, the decoding device does not need to perform separate quality calculation operations, thereby reducing power consumption during the decoding process and enabling more consistent quality evaluation according to criteria defined during the encoding stage.

[0381] For example, the encoding device can generate at least one QM (Quality Metic) SEI (supplemental enhancement information) message included in at least one picture unit based on quality metric information. For example, the processor of the encoding device can generate at least one QM SEI message included in at least one picture unit based on quality metric information.

[0382] For example, the processor of the encoding device can generate a QM SEI message based on the acquired quality indicator, including information on whether a definition of the quality indicator exists, information on at least one sub-layer for the picture unit, information on whether an average quality exists, average quality indicator information, information on the number of quality indicator entries, quality indicator component information, quality indicator type information, and information on whether there is a quality gain.

[0383] The QM SEI message may be the same as the QM SEI message described in operation S520 of FIG. 5. The description of the QM SEI message may be replaced with the description of the QM SEI message described in operation S520 of FIG. 5.

[0384] For example, the encoding device can generate information about at least one sublayer for the picture unit based on the acquired quality indicator.

[0385] For example, information regarding at least one sub-layer for a picture unit may include information on the maximum number of at least one sub-layer and information on the number of pictures included in at least one sub-layer.

[0386] Information on the maximum number of sub-layers can represent the maximum number of temporal sub-layers used to signal average quality metric information for CLVS in units of temporal sub-layers.

[0387] Information regarding the maximum number of sublayers may take various forms and be expressed by various names. For example, information regarding the maximum number of sublayers may be a syntax element or a syntax structure containing one or more syntax elements. For example, information regarding the maximum number of sublayers may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, information regarding the maximum number of sublayers may include, but is not limited to, qm_max_sublayers_minus1. For example, if information regarding the maximum number of sublayers is expressed as qm_max_sublayers_minus1, the value obtained by adding 1 to qm_max_sublayers_minus1 may represent the maximum number of temporal sublayers for signaling average quality metric information within CLVS.

[0388] The picture count information included in the sub-layer can represent the number of valid pictures corresponding to each temporal sub-layer in order to calculate the value of the average quality indicator information at the temporal sub-layer level within CLVS.

[0389] For example, information on the number of pictures included in at least one sublayer may represent the number of at least one picture having an identifier that is less than or equal to the identifier corresponding to that sublayer. That is, information on the number of pictures included in a temporal sublayer may represent the number of all pictures having a temporal identifier that is equal to or smaller than the temporal identifier (temporal ID) of that sublayer. In other words, information on the number of pictures included in a sublayer may represent the number of pictures referenced to calculate the average quality metric for each sublayer.

[0390] The picture count information included in the sublayer may take various forms and may be expressed by various names. For example, the picture count information included in the sublayer may be a syntax element or a syntax structure containing one or more syntax elements. For example, the picture count information included in the sublayer may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, the picture count information included in the sublayer may include qm_num_pic_in_tid[k], but is not limited thereto.

[0391] For example, if the information regarding the number of pictures included in a sublayer is represented by qm_num_pic_in_tid[k], then if such picture count information exists, qm_num_pic_in_tid[k] may represent the number of pictures included in TestPicList for which the temporal sublayer identifier is k or less. Here, k may represent the temporal ID of the temporal sublayer. For example, the value of qm_num_pic_in_tid[k] ranges from 0 to 2 32 - It can be in the range of up to 2 (inclusive).

[0392] For example, information on the number of pictures included in at least one sublayer can be generated based on the number of sublayers according to the information on the maximum number of sublayers.

[0393] For example, information on the number of pictures included in at least one sublayer can be generated repeatedly as many times as the number of sublayers determined by the information on the maximum number of sublayers generated in the QM SEI message. That is, based on the number of sublayers determined by the information on the maximum number of sublayers generated in the QM SEI message, the number of pictures having a temporal ID below that layer can be sequentially generated for each sublayer layer. Through this, the decoding device can generate information on the number of pictures in each sublayer, indicating the number of accumulated pictures included up to that layer. The decoding device can generate information on the number of pictures included in each sublayer based on the number of accumulated pictures.

[0394] The encoding device can generate average quality indicator information corresponding to at least one sublayer for the picture unit from the QM SEI message.

[0395] Average quality metric information may take various forms and may be represented by various names. For example, average quality metric information may be a syntax element or a syntax structure containing one or more syntax elements. For example, average quality metric information may include a flag consisting of one bit, an indicator consisting of two or more bits or a variable-length bit, or an unsigned integer consisting of two or more bits or a variable-length bit. For example, average quality metric information may include qm_clvs_metric_value[i], but is not limited thereto.

[0396] For example, when average quality metric information is represented as qm_clvs_metric_value[k][i][c], qm_clvs_metric_value[k][i][c] may represent the average value of the i-th quality metric entry for the c-th component, calculated for all pictures in the CLVS where the temporal ID is k or less. Here, index k represents the temporal ID of the temporal sub-layer, index i is an index identifying the type of quality metric entry and can be calculated based on the number of quality metric entries, and index c may be a component index determined based on quality metric component information indicating whether the quality metric is composed of a single component or three color components (Y, U, V, etc.).

[0397] For example, average quality indicator information can be generated by averaging the quality indicator values ​​of at least one picture having an identifier that is less than or equal to the identifier corresponding to that sublayer, based on picture count information included in at least one sublayer.

[0398] For example, information on the maximum number of sublayers and the number of pictures included in the sublayers can be generated based on the existence of average quality indicator information within the QM SEI message. That is, if average quality indicator information exists within the QM SEI message, the encoding device can generate information on the maximum number of sublayers and the number of pictures included in the sublayers.

[0399] According to the present disclosure, the encoding device may generate a layer-specific average quality indicator value based on a temporal sub-layer, rather than generating a single average quality indicator value for the entire CLVS. Additionally, the encoding device may generate independent average quality indicator values ​​for each sub-layer corresponding to a temporal ID and generate a QM SEI message containing these values.

[0400] The encoding device can encode video information (S620).

[0401] For example, the processor of the encoding device can generate a supplemental enhancement information (SEI) message containing quality metric information and encode the pictures and image information containing the SEI message.

[0402] An encoding device can encode an image or image information. Here, the image may include a still image or a video, and the image information may refer to pixel data of the image, object information, additional information (e.g., SEI messages, etc.), or a combination thereof.

[0403] An encoding device can generate a bitstream by performing an encoding process on an input image or image information. At this time, the encoding process may include one or more steps such as prediction, transformation, quantization, and entropy encoding, and the generated bitstream can be used as data for reconstructing the image or image information by a decoding device.

[0404] Additionally, the encoding device according to the present disclosure may encode a QM SEI message including additional information regarding the quality of an image, and the QM SEI message may include information on whether a definition of a quality indicator exists, information regarding at least one sub-layer for a picture unit, information on whether an average quality exists, average quality indicator information, information on the number of quality indicator entries, quality indicator component information, quality indicator type information, and information on whether there is a quality gain.

[0405] Accordingly, the present disclosure can be applied not only to general encoding structures for encoding images or image information, but also to encoding methods of QM SEI messages including quality indicator information and related additional information.

[0406] For example, image information may include SEI messages. SEI messages may convey specific types of information that assist in processes related to the decoding, display, or other processing purposes of image information. Here, SEI messages may include QM SEI messages that provide quality indicator information for a picture or set of pictures.

[0407] Specifically, a QM SEI message may include one or more quality indicator information representing the quality of a single picture, the average quality of a set of pictures, or quality gain. In particular, one or more QM SEI messages may each provide quality indicator information for one or more pictures associated with the respective QM SEI message. In other words, among the multiple QM SEI messages, one QM SEI message may be associated with the picture currently being decoded. Additionally, one QM SEI message may exist within the current picture and convey quality indicator information for that picture.

[0408] In this way, the processor of the encoding device can calculate quality indicator information for one or more pictures and construct a QM SEI message corresponding to the result, and encode image information including said QM SEI message.

[0409] Video information encoded according to the encoding method (S600) described above can be output in the form of a bitstream. In other words, the bitstream can be generated based on the video information encoded according to the encoding method (S600) described above.

[0410] A bitstream generated based on video information encoded according to the encoding method (S600) described above can be stored on a computer-readable storage medium.

[0411] A bitstream generated based on video information encoded according to the encoding method (S600) described above can be transmitted through a transmission unit and / or a transmission medium.

[0412] FIG. 7 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0413] As illustrated in FIG. 7, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0414] The above encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data to generate a bitstream and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.

[0415] The bitstream may be generated by a video encoding method and / or encoding device to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0416] The streaming server transmits multimedia data to a user device based on a user request through a web server, and the web server can act as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server can perform the role of controlling commands and responses between each device within the content streaming system.

[0417] The streaming server can receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store the bitstream for a certain period of time.

[0418] Examples of the above user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0419] Each server within the above-mentioned content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0420] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.

[0421] An embodiment according to the present disclosure can be used to encode / decode images.

Claims

1. In a video decoding method performed by a decoding device, A step of acquiring a QM (quality metric) SEI (supplemental enhancement information) message contained in at least one picture unit; and It includes the step of obtaining quality indicator information based on the above QM SEI message, and The above quality indicator information is, A video decoding method characterized by including information regarding at least one sublayer for the picture unit and average quality indicator information corresponding to the at least one sublayer.

2. In Paragraph 1, An image decoding method characterized in that the information regarding the at least one sublayer includes information on the maximum number of the at least one sublayer and information on the number of pictures included in the at least one sublayer.

3. In Paragraph 2, A video decoding method characterized in that the picture count information included in at least one sublayer indicates at least one picture count having an identifier smaller than or equal to the identifier corresponding to the sublayer.

4. In Paragraph 3, The above average quality indicator information is, A video decoding method characterized by averaging the quality indicator values ​​of at least one picture having an identifier smaller than or equal to the identifier corresponding to the sublayer, based on picture count information included in at least one sublayer.

5. In Paragraph 2, An image decoding method characterized in that the picture count information included in at least one sublayer is obtained based on the number of sublayers according to the maximum number information.

6. In Paragraph 2, The above quality indicator information further includes information on whether average quality exists, and A video decoding method characterized by obtaining the maximum number information and the picture number information based on the information regarding the existence of the average quality.

7. In a video encoding method performed by an encoding device, A step of generating at least one QM (Quality Metic) SEI (supplemental enhancement information) message included in at least one picture unit; and The method includes the step of encoding image information including at least one QM SEI message, and Quality indicator information, A video encoding method characterized by including information regarding at least one sublayer for the picture unit, and average quality indicator information corresponding to the at least one sublayer.

8. In Paragraph 7, A video encoding method characterized in that the information regarding the at least one sublayer includes information on the maximum number of the at least one sublayer and information on the number of pictures included in the at least one sublayer.

9. In Paragraph 8, A video encoding method characterized in that the picture count information included in at least one sublayer indicates at least one picture count having an identifier smaller than or equal to the identifier corresponding to the sublayer.

10. In Paragraph 9, The above average quality indicator information is, A video encoding method characterized by generating a quality indicator value of at least one picture having an identifier smaller than or equal to an identifier corresponding to the sublayer, based on picture count information included in at least one sublayer.

11. In Paragraph 8, A video encoding method characterized in that the picture count information included in at least one sublayer is generated based on the number of sublayers according to the maximum number information.

12. In Paragraph 8, A video encoding method characterized by generating the maximum number information and the picture number information based on the fact that the above average quality indicator information exists within the above QM SEI message.

13. In a computer-readable storage medium for storing a bitstream, At least one QM (Quality Metic) SEI (supplemental enhancement information) message included in at least one picture unit is generated based on quality indicator information, and a bitstream generated based on image information including said at least one QM SEI message is stored in a computer-readable storage medium. The above quality indicator information is, A storage medium characterized by including information regarding at least one sublayer for the picture unit, and average quality indicator information corresponding to the at least one sublayer.

14. A method for transmitting data regarding an image, wherein the method comprises the step of acquiring image information, wherein the image information includes at least one QM (Quality Metic) SEI (supplemental enhancement information) message included in at least one picture unit; and The method includes the step of transmitting the data including the above image information, Quality indicator information, A bitstream transmission method characterized by including information regarding at least one sublayer for the picture unit, and average quality indicator information corresponding to the at least one sublayer.

Images