Image encoding / decoding method and device, and recording medium having bitstream stored therein

Abstract

An image decoding method and device according to the present disclosure may: receive a bitstream comprising an encoded video picture; and reconstruct the encoded video picture included in the bitstream. Here, the bitstream may comprise a generative face video supplementary enhancement information (SEI) message. The generative face video SEI message may comprise at least one of a reference picture flag indicating whether the currently decoded output picture corresponds to a reference picture and a matrix prediction flag indicating whether matrix element difference value information is present.

Description

Video encoding / decoding method and device, and a recording medium storing a bitstream

[0001] The present invention relates to a video encoding / decoding method and apparatus, and a recording medium storing a bitstream.

[0002] Recently, the demand for high-resolution, high-quality video, such as HD (High Definition) and UHD (Ultra High Definition) video, has been increasing across various application fields, and accordingly, high-efficiency video compression technologies are being discussed.

[0003] Various image compression technologies exist, such as inter-prediction technology that predicts pixel values ​​in the current picture from previous or subsequent pictures, intra-prediction technology that predicts pixel values ​​in the current picture using pixel information within the current picture, and entropy coding technology that assigns short codes to values ​​with high frequency and long codes to values ​​with low frequency; by utilizing these image compression technologies, image data can be effectively compressed for transmission or storage.

[0004] The present disclosure provides a method and apparatus for configuring a generative face video SEI (Supplementary Enhancement Information) message including generative face video information.

[0005] The present disclosure provides a method and apparatus for signaling a generated face video SEI message.

[0006] A video decoding method and apparatus according to the present disclosure receive a bitstream including an encoded video picture and can restore the encoded video picture included in the bitstream. The bitstream may include a generated face video SEI (Supplementary Enhancement Information) message.

[0007] In the image decoding method and apparatus according to the present disclosure, the generated face video SEI message may include at least one of a reference picture flag indicating whether the currently decoded output picture corresponds to a reference picture, or a matrix prediction flag indicating whether matrix element difference value information exists.

[0008] In the image decoding method and apparatus according to the present disclosure, the generated face video SEI message can be obtained from a network abstraction layer (NAL) unit of the bitstream.

[0009] In the image decoding method and apparatus according to the present disclosure, when the value of the matrix prediction flag is 0, it may indicate that matrix element value information exists, and when the value of the matrix prediction flag is 1, it may indicate that matrix element difference value information exists. Here, the matrix element value information may include at least one of matrix element integer value information or matrix element decimal value information, and the matrix element difference value information may include at least one of matrix element integer difference value information or matrix element decimal difference value information.

[0010] In the image decoding method and apparatus according to the present disclosure, the generated face video SEI message may include matrix precision information indicating the precision of a matrix element, and the matrix precision information may be signaled based on the matrix prediction flag.

[0011] In the image decoding method and apparatus according to the present disclosure, when the value of the matrix prediction flag is 0, the matrix precision information may be signaled, and when the value of the matrix prediction flag is 1, the matrix precision information may not be signaled regardless of the value of the reference picture flag.

[0012] In the image decoding method and apparatus according to the present disclosure, the generated face video SEI message may include matrix type count information indicating the number of matrix types, and the matrix type count information may be signaled based on the matrix prediction flag.

[0013] In the image decoding method and apparatus according to the present disclosure, when the value of the matrix prediction flag is 0, the matrix type count information may be signaled, and when the value of the matrix prediction flag is 1, the matrix type count information may not be signaled regardless of the value of the reference picture flag.

[0014] In the image decoding method and apparatus according to the present disclosure, the generated face video SEI message may include matrix type information indicating the type of matrix included in the generated face video SEI message, and the matrix type information may be signaled based on at least one of the matrix prediction flag or the matrix type count information.

[0015] In the image decoding method and apparatus according to the present disclosure, when the value of the matrix prediction flag is 0, the matrix type information may be signaled, and when the value of the matrix prediction flag is 1, the matrix type information may not be signaled regardless of the value of the reference picture flag.

[0016] In the image decoding method and apparatus according to the present disclosure, the generated face video SEI message may include matrix-related information by matrix type, and the matrix-related information by matrix type may be signaled based on at least one of the matrix prediction flag or the matrix type information. Here, the matrix-related information may include at least one of matrix count information indicating the number of matrices by matrix type, matrix width information indicating the width of the matrix by matrix type, or matrix height information indicating the height of the matrix by matrix type.

[0017] In the image decoding method and apparatus according to the present disclosure, when the value of the matrix prediction flag is 0, matrix-related information for each matrix type may be signaled, and when the value of the matrix prediction flag is 1, matrix-related information for each matrix type may not be signaled regardless of the value of the reference picture flag.

[0018] In the image decoding method and apparatus according to the present disclosure, when the value of the matrix type information is greater than or equal to 7, the matrix count information, the matrix width information, and the matrix height information may be signaled, and when the value of the matrix type information is 2 or 3, the matrix width information and the matrix height information may be signaled, and the matrix count information may not be signaled.

[0019] A video encoding method and apparatus according to the present disclosure may receive a video picture to be encoded, encode the received video picture to generate video information regarding the video picture, generate a generative face video SEI message, and generate a bitstream including the video information and the generative face video SEI message. The generative face video SEI message may include at least one of a reference picture flag indicating whether the currently decoded output picture corresponds to a reference picture, or a matrix prediction flag indicating whether matrix element difference value information exists. The generative face video SEI message may be encoded in a network abstraction layer (NAL) unit of the bitstream.

[0020] A computer-readable digital storage medium is provided that stores encoded video / image information that causes an image decoding method to be performed by a decoding device according to the present disclosure.

[0021] A computer-readable digital storage medium is provided that stores video / image information generated according to the image encoding method according to the present disclosure.

[0022] A method and apparatus for transmitting video / image information generated according to the image encoding method according to the present disclosure are provided.

[0023] According to the present disclosure, by defining a generative face video SEI message, information regarding face parameters for generating a face image, or information regarding a neural network for face parameter transformation and output image generation, can be easily verified.

[0024] According to the present disclosure, by removing redundant operations within a supplemental enhancement information (SEI) message, unnecessary operations in a video encoding / decoding device can be prevented, thereby increasing the utility of the supplemental enhancement information SEI message.

[0025] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0026] FIG. 2 shows a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and to which encoding of a video / image signal is performed.

[0027] FIG. 3 shows a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and to which decoding of a video / image signal is performed.

[0028] FIG. 4 illustrates a method for restoring a video picture performed in a decoding device (300) according to the present disclosure.

[0029] FIG. 5 illustrates a schematic configuration of a decoding device (300) that performs a method for restoring a video picture according to the present disclosure.

[0030] FIG. 6 illustrates a method for generating a bitstream performed in an encoding device (200) according to the present disclosure.

[0031] FIG. 7 illustrates a schematic configuration of an encoding device (200) that performs a method for generating a bitstream according to the present disclosure.

[0032] FIG. 8 shows an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0033] The present disclosure is susceptible to various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals have been used for similar components in the description of each drawing.

[0034] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0035] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0036] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the disclosure. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “having” are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037] The present disclosure relates to video / video coding. For example, the methods / embodiments disclosed herein may be applied to methods disclosed in the VVC (versatile video coding) standard. Additionally, the methods / embodiments disclosed herein may be applied to methods disclosed in the EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard), or next-generation video / video coding standards (e.g., H.267 or H.268).

[0038] This specification presents various embodiments regarding video / image coding, and unless otherwise noted, said embodiments may be performed in combination with one another.

[0039] In this specification, "video" may refer to a set of images over time. "Picture" generally refers to a unit representing a single image of a specific time period, and "slice" or "tile" is a unit that constitutes a part of a picture in coding. A slice or tile may contain one or more coding tree units (CTUs). A picture may consist of one or more slices or tiles. A tile is a rectangular area composed of multiple CTUs within a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs having a height equal to the height of the picture and a width specified by the syntax requirements of the picture parameter set. A tile row is a rectangular area of ​​CTUs having a height specified by the picture parameter set and a width equal to the width of the picture. CTUs within a tile are arranged continuously according to the CTU raster scan, whereas tiles within a picture may be arranged continuously according to the tile's raster scan. A single slice may include an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of a picture that can be exclusively contained in a single NAL unit. Meanwhile, a single picture may be divided into two or more subpictures. A subpicture may be a rectangular area of ​​one or more slices within a picture.

[0040] A pixel, or pel, can refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. A sample generally represents a pixel or its value, and it may represent only the pixel / pixel value of the luminance (luma) component or only the pixel / pixel value of the chroma component.

[0041] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. A unit may include one luminance block and two chroma (e.g., cb, cr) blocks. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0042] In this specification, "A or B" may mean "only A," "only B," or "both A and B." Alternatively, in this specification, "A or B" may be interpreted as "A and / or B." For example, in this specification, "A, B or C" may mean "only A," "only B," "only C," or "any combination of A, B and C."

[0043] A slash ( / ) or a comma used in this specification may mean "and / or." For example, "A / B" may mean "A and / or B." Accordingly, "A / B" may mean "only A," "only B," or "both A and B." For example, "A, B, C" may mean "A, B or C."

[0044] In this specification, "at least one of A and B" may mean "only A," "only B," or "both A and B." Additionally, in this specification, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as synonymous with "at least one of A and B."

[0045] Additionally, in this specification, "at least one of A, B and C" may mean "only A," "only B," "only C," or "any combination of A, B and C." Also, "at least one of A, B or C" or "at least one of A, B and / or C" may mean "at least one of A, B and C."

[0046] Additionally, parentheses used in this specification may mean "for example." Specifically, where indicated as "prediction (intra-prediction)," "intra-prediction" may be proposed as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra-prediction," and "intra-prediction" may be proposed as an example of "prediction." Furthermore, even where indicated as "prediction (i.e., intra-prediction)," "intra-prediction" may be proposed as an example of "prediction."

[0047] Technical features described individually within a single drawing in this specification may be implemented individually or simultaneously.

[0048] FIG. 1 illustrates a video / image coding system according to the present disclosure.

[0049] Referring to FIG. 1, the video / image coding system may include a first device (source device) (10) and a second device (receiving device) (20).

[0050] A source device (10) can transmit encoded video / image information or data in the form of a file or streaming to a receiving device (20) via a digital storage medium or network. The source device (10) may include a video source (11), an encoding device (12), and a transmission unit (13). The receiving device (20) may include a receiving unit (21), a decoding device (22), and a renderer (23). The encoding device (12) may be called a video / image encoding device, and the decoding device (22) may be called a video / image decoding device. A transmitter may be included in the encoding device. A receiver may be included in the decoding device. The renderer (23) may include a display unit, and the display unit may be composed of a separate device or an external component.

[0051] A video source (11) can acquire video / image through a process of capturing, synthesizing, or generating video / image. The video source (11) may include a video / image capture device and / or a video / image generation device. The video / image capture device may include one or more cameras, a video / image archive containing previously captured video / image, etc. The video / image generation device may include a computer, a tablet, a smartphone, etc., and can generate video / image (electronically). For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0052] The encoding device (12) can encode an input video / image. The encoding device (12) can perform a series of procedures such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0053] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit (21) of the receiving device (20) via a digital storage medium or network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit (21) can receive / extract the bitstream and transmit it to a decoding device (22).

[0054] The decoding device (22) can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding device (12).

[0055] The renderer (23) can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0056] FIG. 2 shows a schematic block diagram of an encoding device to which an embodiment of the present disclosure can be applied and to which encoding of a video / image signal is performed.

[0057] Referring to FIG. 2, the encoding device (200) may be configured to include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-predictor (221) and an intra-predictor (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit (250) may be referred to as a reconstructor or a reconstructed block generator. The above-described image segmentation unit (210), prediction unit (220), residual processing unit (230), entropy encoding unit (240), addition unit (250), and filtering unit (260) may be configured by one or more hardware components (e.g., an encoding device chipset or processor) according to the embodiment. Additionally, the memory (270) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (270) as an internal / external component.

[0058] The image segmentation unit (210) can divide an input image (or picture, frame) input to an encoding device (200) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure.

[0059] For example, a single coding unit may be divided into multiple coding units with a deeper depth based on a quad tree structure, a binary tree structure, and / or a terrestrial structure. In this case, for example, the quad tree structure may be applied first and the binary tree structure and / or terrestrial structure may be applied later. Alternatively, the binary tree structure may be applied before the quad tree structure. A coding procedure according to the present specification may be performed based on a final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively divided into coding units of a lower depth so that a coding unit of the optimal size may be used as the final coding unit. Here, the term "coding procedure" may include procedures such as prediction, transformation, and restoration described below.

[0060] As another example, the processing unit may further include a Prediction Unit (PU) or a Transform Unit (TU). In this case, the Prediction Unit and the Transform Unit may each be divided or partitioned from the aforementioned final coding unit. The Prediction Unit may be a unit for sample prediction, and the Transform Unit may be a unit for deriving transformation coefficients and / or a unit for deriving a residual signal from transformation coefficients.

[0061] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.

[0062] The encoding device (200) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (prediction block, prediction sample array) output from an inter prediction unit (221) or an intra prediction unit (222) from an input video signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (232). In this case, the unit that subtracts the prediction signal (prediction block, prediction sample array) from the input video signal (original block, original sample array) within the encoding device (200) may be called a subtraction unit (231).

[0063] The prediction unit (220) performs a prediction for a block to be processed (hereinafter referred to as the current block) and can generate a predicted block containing prediction samples for the current block. The prediction unit (220) can determine whether intra prediction is applied or inter prediction is applied at the current block or CU level. The prediction unit (220) can generate various information regarding the prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (240). The information regarding the prediction can be encoded by the entropy encoding unit (240) and output in the form of a bitstream.

[0064] The intra prediction unit (222) can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or at a certain distance from the current block depending on the prediction mode. In intra prediction, the prediction modes may include one or more non-directional modes and multiple directional modes. The non-directional mode may include at least one DC mode or a planar mode. The directional mode may include 33 directional modes or 65 directional modes depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, more or fewer directional modes may be used. The intra prediction unit (222) may determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.

[0065] The inter prediction unit (221) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different. The above temporal surrounding blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc., and the reference picture containing the above temporal surrounding blocks may be referred to as a collocated picture (colPic). For example, the inter prediction unit (221) may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the inter prediction unit (221) may use the motion information of surrounding blocks as motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vectors of surrounding blocks are used as motion vector predictors, and the motion vector of the current block can be indicated by signaling the motion vector difference.

[0066] The prediction unit (220) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for prediction of a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called a combined inter and intra prediction (CIIP) mode. Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in screen content coding (SCC) for games. IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this specification. The palette mode can be viewed as an example of intra coding or intra prediction. When the palette mode is applied, sample values ​​within the picture can be signaled based on information regarding the palette table and palette index. The prediction signal generated through the prediction unit (220) can be used to generate a restoration signal or to generate a residual signal.

[0067] The transformation unit (232) can generate transform coefficients by applying a transformation technique to a residual signal. For example, the transformation technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT). Here, GBT refers to a transformation obtained from a graph when the relationship information between pixels is represented as a graph. CNT refers to a transformation obtained based on a prediction signal generated using all previously restored pixels. Additionally, the transformation process may be applied to a pixel block of the same size in a square, or to a block of variable size that is not square.

[0068] The quantization unit (233) quantizes the transformation coefficients and transmits them to the entropy encoding unit (240), and the entropy encoding unit (240) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (233) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients.

[0069] The entropy encoding unit (240) can perform various encoding methods such as exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (240) may encode information required for video / image restoration (e.g., values ​​of syntax elements, etc.) together or separately, in addition to quantized transform coefficients.

[0070] Encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream at the level of a Network Abstraction Layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an Adaptation Parameter Set (APS), a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), or a Video Parameter Set (VPS). Additionally, the video / image information may further include general constraint information. In this specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream. The bitstream may be transmitted over a network or stored on a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits the signal output from the entropy encoding unit (240) and / or a storage unit (not shown) that stores it may be configured as internal / external elements of the encoding device (200), or the transmission unit may be included in the entropy encoding unit (240).

[0071] Quantized transformation coefficients output from the quantization unit (233) can be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit (234) and the inverse transformation unit (235). An adder (250) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the inter-prediction unit (221) or the intra-prediction unit (222). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (250) may be called a reconstruction unit or a reconstruction block generation unit. The generated restoration signal can be used for intra prediction of the next block to be processed within the current picture, and can also be used for inter prediction of the next picture after filtering as described below. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.

[0072] The filtering unit (260) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (260) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (270), specifically in the DPB of memory (270). The various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (260) can generate various information regarding filtering and transmit it to the entropy encoding unit (240). The information regarding filtering can be encoded in the entropy encoding unit (240) and output in the form of a bitstream.

[0073] The modified restored picture transmitted to the memory (270) can be used as a reference picture in the inter-prediction unit (221). Through this, when inter-prediction is applied, the encoding device can avoid prediction mismatches between the encoding device (200) and the decoding device, and can also improve encoding efficiency.

[0074] The DPB of the memory (270) can store the modified restored picture to be used as a reference picture in the inter-prediction unit (221). The memory (270) can store motion information of blocks from which motion information is derived (or encoded) within the current picture and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit (221) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (270) can store restoration samples of the blocks restored within the current picture and transmit them to the intra-prediction unit (222).

[0075] FIG. 3 shows a schematic block diagram of a decoding device to which an embodiment of the present disclosure can be applied and to which decoding of a video / image signal is performed.

[0076] Referring to FIG. 3, the decoding device (300) may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-predictor (332) and an intra-predictor (331). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321).

[0077] The aforementioned entropy decoding unit (310), residual processing unit (320), prediction unit (330), addition unit (340), and filtering unit (350) may be configured by a single hardware component (e.g., a decoding device chipset or processor) according to an embodiment. Additionally, the memory (360) may include a DPB (decoded picture buffer) and may be configured by a digital storage medium. The hardware component may further include the memory (360) as an internal / external component.

[0078] When a bitstream containing video / image information is input, the decoding device (300) can restore the image in correspondence with the process in which the video / image information is processed by the encoding device of FIG. 2. For example, the decoding device (300) can derive units / blocks based on block division information obtained from the bitstream. The decoding device (300) can perform decoding using a processing unit applied by the encoding device. Accordingly, the processing unit for decoding may be a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a binary tree structure. One or more conversion units may be derived from the coding unit. And, the restored image signal decoded and output through the decoding device (300) can be played back through a playback device.

[0079] The decoding device (300) can receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal can be decoded through an entropy decoding unit (310). For example, the entropy decoding unit (310) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The decoding device can decode the picture based on information regarding the parameter sets and / or the general constraint information. The signaling / receiving information and / or syntax elements described below in this specification may be decoded through the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (310) can decode information within the bitstream based on coding methods such as exponential chord coding, CAVLC, or CABAC, and output the values ​​of syntax elements required for image restoration and the quantized values ​​of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in the bitstream, determine a context model using information on the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information on symbols / bins decoded in the previous step, predict the probability of occurrence of the bin according to the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin after determining the context model. Among the information decoded in the entropy decoding unit (310), information regarding prediction is provided to the prediction unit (inter prediction unit (332) and intra prediction unit (331)), and the residual value for which entropy decoding was performed in the entropy decoding unit (310), i.e., quantized transformation coefficients and related parameter information, can be input to the residual processing unit (320). The residual processing unit (320) can derive residual signals (residual blocks, residual samples, residual sample array). Additionally, among the information decoded in the entropy decoding unit (310), information regarding filtering can be provided to the filtering unit (350). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (300), or the receiving unit may be a component of the entropy decoding unit (310).

[0080] Meanwhile, the decoding device according to the present specification may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoding device (video / image / picture information decoding device) and a sample decoding device (video / image / picture sample decoding device). The information decoding device may include the entropy decoding unit (310), and the sample decoding device may include at least one of the inverse quantization unit (321), inverse transform unit (322), adder (340), filtering unit (350), memory (360), inter prediction unit (332), and intra prediction unit (331).

[0081] In the inverse quantization unit (321), the quantized transformation coefficients can be inversely quantized to output transformation coefficients. The inverse quantization unit (321) can rearrange the quantized transformation coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit (321) can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain transformation coefficients.

[0082] In the inverse conversion unit (322), the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).

[0083] The prediction unit (320) can perform a prediction for the current block and generate a predicted block containing prediction samples for the current block. The prediction unit (320) can determine whether an intra prediction or an inter prediction is applied to the current block based on information regarding the prediction output from the entropy decoding unit (310), and can determine a specific intra / inter prediction mode.

[0084] The prediction unit (320) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit (320) may apply intra prediction or inter prediction for prediction of a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called a combined inter and intra prediction (CIIP) mode. Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content video / video coding, such as in games, such as SCC (screen content coding). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this specification. The palette mode can be viewed as an example of intra coding or intra prediction. When palette mode is applied, information regarding the palette table and palette index can be included in the above video / image information and signaled.

[0085] The intra prediction unit (331) can predict the current block by referring to samples within the current picture. The referenced samples may be located in the neighborhood of the current block according to the prediction mode, or may be located at a certain distance from the current block. In intra prediction, the prediction modes may include one or more non-directional modes and a plurality of directional modes. The intra prediction unit (331) may determine the prediction mode applied to the current block by using the prediction mode applied to the neighboring blocks.

[0086] The inter prediction unit (332) can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit (332) may construct a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter-prediction can be performed based on various prediction modes, and information regarding the prediction may include information indicating the inter-prediction mode for the current block.

[0087] The adder (340) can generate a restoration signal (restoration picture, restoration block, restoration sample array) by adding the acquired residual signal to the prediction signal (prediction block, prediction sample array) output from the prediction unit (including the inter prediction unit (332) and / or the intra prediction unit (331)). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the prediction block can be used as the restoration block.

[0088] The addition unit (340) may be called a restoration unit or a restoration block generation unit. The generated restoration signal may be used for intra-predicting the next block to be processed within the current picture, may be output after filtering as described below, or may be used for inter-predicting the next picture. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0089] The filtering unit (350) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (350) can generate a modified restored picture by applying various filtering methods to the restored picture, and can transmit the modified restored picture to memory (360), specifically to the DPB of memory (360). The various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0090] The (modified) restored picture stored in the DPB of the memory (360) can be used as a reference picture in the inter prediction unit (332). The memory (360) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter prediction unit (332) to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (360) can store restoration samples of blocks restored within the current picture and transmit them to the intra prediction unit (331).

[0091] In this specification, the embodiments described in the filtering unit (260), inter prediction unit (221), and intra prediction unit (222) of the encoding device (200) may be applied to the filtering unit (350), inter prediction unit (332), and intra prediction unit (331) of the decoding device (300) in the same or corresponding manner.

[0092] FIG. 4 illustrates a method for restoring a video picture performed in a decoding device (300) according to the present disclosure.

[0093] A bitstream containing an encoded video picture can be received (S400).

[0094] The encoded video picture of the bitstream can be restored (S410).

[0095] Video information regarding an encoded video picture can be extracted from the bitstream. Based on the extracted video information, the encoded video picture can be restored.

[0096] The bitstream may include generative face video information. The generative face video information may be configured in the Supplementary Enhancement Information (SEI) message of the bitstream. In this case, the SEI message may be named a Generative Face Video (GFV) SEI message.

[0097] Generative face video information may include information regarding face parameters for generating a face image, or information regarding a neural network for face parameter transformation and output image generation. More specifically, the generative face video information may include information regarding a face parameter translator neural network (TranslatorNN()) that can be used to convert face parameters of various formats signaled to a decoding device into face parameters of a specific format supported by the decoding system. Additionally, the generative face video information may include information regarding a face picture generator neural network (GenerativeNN()) that can be used to generate a final output picture based on face parameters converted to a specific format through the face parameter translator neural network and a previously decoded output picture.

[0098] Table 1 is an example of a generated face video SEI message included in a bitstream.

[0099] generative_face_video ( payloadSize ) {Descriptorgfv_idue(v)gfv_cntue(v)if( gfv_cnt = = 0 )gfv_base_pic_flag / * indicate if the current decoded output picture is a base picture * / u(1)if( gfv_base_pic_flag ) { / * specify TranslatorNN( ) * / gfv_nn_present_flagu(1)if( gfv_nn_present_flag ) {gfv_nn_mode_idcue(v)if( gfv_nn_mode_idc = = 1 ) {while( !byte_aligned( ) )gfv_nn_alignment_zero_bit_au(1)gfv_nn_tag_urist(v)gfv_nn_urist(v)}}gfv_chroma_key_info_present_flagu(1)if( gfv_chroma_key_info_present_flag ) {for( c = 0; c < 3; c++ ) {gfv_chroma_key_value_present_flag[ c ]u(1)if( gfv_chroma_key_value_present_flag[ c ] )gfv_chroma_key_value[ c ]u(8)}for( i = 0; i < 2;i++ ) {gfv_chroma_key_thr_present_flag[ i ]u(1)if( gfv_chroma_key_thr_present_flag[ i ] )gfv_chroma_key_thr_value[ i ]ue(v)}}} elsegfv_drive_pic_fusion_flag / * indicate if DrivePicture is input to GenerativeNN( ) * / u(1)gfv_low_confidence_face_parameter_flagu(1)gfv_coordinate_present_flagu(1)if( gfv_coordinate_present_flag ) {gfv_kps_pred_flagu(1)if( gfv_base_pic_flag | | !gfv_kps_pred_flag ) {gfv_coordinate_precision_factor_minus1ue(v)gfv_num_kps_minus1ue(v)gfv_coordinate_z_present_flagu(1)if(gfv_coordinate_z_present_flag )gfv_coordinate_z_max_value_minus1ue(v)}for( i = 0; i <= gfv_num_kps_minus1;i++ ) {if( !gfv_kps_pred_flag ) {gfv_coordinate_x_abs[ i ]ue(v)if( gfv_coordinate_x_abs[ i ] > 0 )gfv_coordinate_x_sign_flag[ i ]u(1)gfv_coordinate_y_abs[ i ]ue(v)if( gfv_coordinate_y_abs[ i ] > 0 )gfv_coordinate_y_sign_flag[ i ]u(1)if( gfv_coordinate_z_present_flag > 0 ) {gfv_coordinate_z_abs[ i ]ue(v)if( gfv_coordinate_z_abs[ i ] > 0 )gfv_coordinate_z_sign_flag[ i ]u(1)}} else {gfv_coordinate_dx_abs[ i ]ue(v)if( gfv_coordinate_dx_abs[ i ] > 0 )gfv_coordinate_dx_sign_flag[ i ]u(1)gfv_coordinate_dy_abs[ i ]ue(v)if( gfv_coordinate_dy_abs[ i ] > 0 )gfv_coordinate_dy_sign_flag[ i ]u(1)if( gfv_coordinate_z_present_flag ) {gfv_coordinate_dz_abs[ i ]ue(v)if( gfv_coordinate_dz_abs[ i ] > 0 )gfv_coordinate_dz_sign_flag[ i ]u(1)}}}}gfv_matrix_present_flagu(1)if(gfv_matrix_present_flag ) {if( !gfv_base_pic_flag )gfv_matrix_pred_flagu(1)if( gfv_base_pic_flag | | !gfv_matrix_pred_flag ) {gfv_matrix_element_precision_factor_minus1ue(v)gfv_num_matrix_types_minus1ue(v)for( i = 0;i <= num_matrix_types_minus1; i++ ) {gfv_matrix_type_idx[ i ]u(6)if( gfv_matrix_type_idx[ i ] = = 0 | | gfv_matrix_type_idx[ i ] = = 1 ) {if( gfv_coordinate_present_flag )gfv_num_matrices_equal_to_num_kps_flag[ i ]u(1)if(!gfv_num_matrices_equal_to_num_kps_flag[ i ] )gfv_num_matrices_info[ i ]ue(v)} else if( gfv_matrix_type_idx[ i ] = = 2 | | gfv_matrix_type_idx[ i ] = = 3 | |gfv_matrix_type_idx[ i ] >= 7 ) {if( gfv_matrix_type_idx[ i ] >= 7 )gfv_num_matrices_minus1[ i ]ue(v)gfv_matrix_width_minus1[ i ]ue(v)gfv_matrix_height_minus1[ i ]ue(v)} else if( gfv_matrix_type_idx[ i ] >= 4 && gfv_matrix_type_idx[ i ] <= 6 &&!gfv_coordinate_present_flag )gfv_matrix_for_3D_space_flag[ i ]u(1)}}for( i = 0; i <= num_matrix_types_minus1; i++ ) {for( j = 0; j < numMatrices[ i ]; j++ )for( k = 0; k < matrixHeight[ i ]; k++ )for( m = 0; m <matrixWidth[ i ];m++ ) {if( !gfv_matrix_pred_flag ) {gfv_matrix_element_int[ i ][ j ][ k ][ m ]ue(v)gfv_matrix_element_dec[ i ][ j ][ k ][ m ]u(v)if( gfv_matrix_element_int[ i][ j ][ k ][ m ] | |gfv_matrix_element_dec[ i ][ j ][ k ][ m ] )gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ]u(1)} else {gfv_matrix_delta_element_int[ i ][ j ][ k ][ m ]ue(v)gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ]ue(v)if( gfv_matrix_delta_element_int[ i][ j ][ k ][ m ] | |gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ] )gfv_matrix_delta_element_sign_flag[ i ][ j ][ k ][ m ]u(1)}}}}if( gfv_nn_present_flag )if( gfv_nn_mode_idc = = 0 ) {while( !byte_aligned( ) )gfv_nn_alignment_zero_bit_bu(1)for( i = 0; more_data_in_payload( ); i++ )gfv_nn_payload_byte[ i ]b(8)}};

[0100] Table 2 is an example of a description of a generated face video SEI message included in a bitstream.

[0101] The generative face video (GFV) SEI message carries facial parameters and indicates a facial parameter translator network, denoted as TranslatorNN( ), that may be used to convert various formats of facial parameters signalled in the SEI message into a particular facial parameter format supported by the decoding system. A face picture generator neural network, denoted as GenerativeNN( ), may be used to generate output pictures using the facial parameters translated into the particular format and previously decoded output pictures.When a picture unit contains a GFV SEI message with a particular gfv_id value and gfv_base_pic_flag equal to 1, the picture in the picture unit is referred to as a base picture for that particular gfv_id value.When a picture unit contains a GFV SEI message with a particular gfv_id value and gfv_base_pic_flag equal to 0, and the picture unit does not contain a GFV SEI message with that particular gfv_id value and gfv_base_pic_flag equal to 1, the picture in the picture unit is referred to as a driving picture for that particular gfv_id value.When a picture unit contains a GFV SEI message with a particular gfv_id value, gfv_base_pic_flag equal to 0, and gfv_drive_pic_fusion_flag equal to 1, and the picture unit does not contain a GFV SEI message with that particular gfv_id value and gfv_base_pic_flag equal to 1, the picture in the picture unit is referred to as a fusion picture for that particular gfv_id value.NOTE 1 - Facial parameters could be determined from source pictures prior to encoding.NOTE 2 - Previously decoded output pictures input to GenerativeNN( ) may be a base picture (a decoded output picture that provides the reference texture from which the face pictures may be generated) and, optionally, a picture that can be fused by GenerativeNN( ) to improve background texture and facial details. When the current picture is not a base picture, the GFV SEI message may be used to generate a face picture based on the previously decoded base picture, the facial parameters conveyed by the GFV SEI message, and, optionally, the current decoded picture for fusion purpose.Use of this SEI message requires the definition of the following variables:Input and output picture width and height in units of luma samples, denoted herein by CroppedWidth and CroppedHeight, respectively.Luma sample array baseCroppedYPic and chroma sample arrays baseCroppedCbPic and baseCroppedCrPic for a decoded output picture, denoted as BasePicture, corresponding to a source base picture.Luma sample array driveCroppedYPic and chroma sample arrays driveCroppedCbPic and driveCroppedCrPic for a decoded output picture, denoted as DrivePicture, corresponding to a source driving picture.Bit depth BitDepth. Y for the luma sample array of the input and output pictures.Bit depth BitDepth C for the chroma sample arrays, if any, of the input and output pictures.A chroma format indicator, denoted herein by ChromaFormatIdc, as described in subclause 7.3.The variables SubWidthC and SubHeightC are derived from ChromaFormatIdc as specified by Table A.gfv_idcontains an identifying number that may be used to identify face feature information and specify a neural network that may be used as TranslatorNN( ). The value of gfv_id shall be in the range of 0 to 2 32 - 2, inclusive. Values of gfv_id from 256 to 511, inclusive, and from 2 31 to 2 32- 2, inclusive, are reserved for future use by ITU-T | ISO / IEC. Decoders conforming to this edition of this document encountering a GFV SEI message with gfv_id in the range of 256 to 511, inclusive, or in the range of 2 31 to 2 32- 2, inclusive, shall ignore the SEI message.gfv_cntspecifies a GFV SEI message instance count value for this gfv_id value within a picture unit.The gfv_cnt of the first GFV SEI message, in decoding order, with a particular value of gfv_id within a picture unit shall be equal to 0. When gfv_cnt assigned to currGfvCnt is greater than 0, a GFV SEI message with the same gfv_id value and gfv_cnt equal to currGfvCnt - 1 shall be present in the same picture unit and precede the current GFV SEI message in decoding order.The value of gfv_cnt shall be in the range of 0 to 65 535, inclusive.gfv_base_pic_flagequal to 1 indicates that the current decoded output picture corresponds to a base picture. gfv_base_pic_flag equal to 0 indicates that the current decoded output picture does not correspond to a base picture or this SEI message does not specify syntax elements for a base picture. When gfv_base_pic_flag is not present, it is inferred to be equal to 0.When a GFV SEI message is the first GFV SEI message, in decoding order, that has a particular gfv_id value within the current CLVS, the value of gfv_base_pic_flag shall be equal to 1.When a GFV SEI message with a particular gfv_id value has gfv_base_pic_flag equal to 1, the base picture for that particular gfv_id value, which is the current cropped decoded picture, remains valid for the current decoded picture and all subsequent decoded pictures of the current layer, in output order, until the end of the current CLVS or up to but excluding the decoded picture that is within the current CLVS, follows the current decoded picture in output order, and is associated with a GFV SEI message having that particular gfv_id value and gfv_base_pic_flag equal to 1, whichever is earlier.gfv_nn_present_flagequal to 1 indicates that a neural network that may be used as a TranslatorNN( ) is contained or indicated by the SEI message.gfv_nn_present_flag equal to 0 indicates that a neural network that may be used as a TranslatorNN( ) is not contained or indicated by the SEI message. When gfv_nn_present_flag is not present, it is inferred to be 0.When gfv_nn_present_flag is equal to 0 and TranslatorNN is referenced in the semantics of the GFV SEI message, the following constraint applies:- If gfv_cnt is equal to 0, there shall be at least one GFV SEI message present in a preceding picture unit in output order in the current CLVS and having the same value of gfv_id as that in the current GFV SEI message and gfv_nn_present_flag equal to 1.- Otherwise (gfv_cnt is greater than 0), there shall be at least one GFV SEI message that is present in either the current picture unit or a preceding picture unit in output order in the current CLVS and has the same value of gfv_id as that in the current GFV SEI message and gfv_nn_present_flag equal to 1.When gfv_nn_present_flag is equal to 0 and TranslatorNN is referenced in the semantics of this SEI message, the following applies for deriving the applicable TranslatorNN:- If gfv_cnt is greater than 0 and there exists one or more preceding GFV SEI messages in decoding order in the current picture unit that has the same value of gfv_id as that in the current GFV SEI message and gfv_nn_present_flag equal to 1, the applicable TranslatorNN is defined by the last preceding GFV SEI message in decoding order in the current picture unit that has the same value of gfv_id as that in the current GFV SEI message and gfv_nn_present_flag equal to 1.- Otherwise, the applicable TranslatorNN is defined by a GFV SEI message that is present in the last preceding picture unit puB in output order in the current CLVS that has the same value of gfv_id as the current GFV SEI message and gfv_nn_present_flag equal to 1.When there are multiple such GFV SEI messages present in the picture unit puB that have the same value of gfv_id as the current GFV SEI message and gfv_nn_present_flag equal to 1, the applicable TranslatorNN is defined by the last of such GFV SEI messages in decoding order.gfv_nn_mode_idc,gfv_nn_alignment_zero_bit_a,gfv_nn_tag_uri,gfv_nn_uri,gfv_nn_alignment_zero_bit_b, andgfv_nn_payload_byte[ i ] specify a neural network that may be used as a TranslatorNN( ). gfv_nn_mode_idc, gfv_nn_alignment_zero_bit_a, gfv_nn_tag_uri, gfv_nn_uri, gfv_nn_alignment_zero_bit_b, and gfv_nn_payload_byte[ i ] have the same syntax and semantics as nnpfc_base_flag, nnpfc_mode_idc, nnpfc_alignment_zero_bit_a, nnpfc_tag_uri, nnpfc_uri, nnpfc_alignment_zero_bit_b, and nnpfc_payload_byte[ i ], respectively.The GFV SEI messages that are present in the same picture unit and have the same values of gfv_id and gfv_cnt shall have the same SEI payload content.gfv_chroma_key_info_present_flagequal to 1 indicates that the syntax elements gfv_chroma_key_value_present_flag[ c ] and gfv_chroma_key_thr_present_flag[ i ] are present and the syntax elements and gfv_chroma_key_value[ c ] and gfv_chroma_key_thr_value[ i ] might be present. gfv_chroma_key_info_present_flag equal to 0 specifies that the syntax elements gfv_chroma_key_value_present_flag[ c ], gfv_chroma_key_thr_present_flag[ i ], gfv_chroma_key_value[ c ], and gfv_chroma_key_thr_value[ i ] are not present.gfv_chroma_key_value_present_flag[ c ] equal to 1 indicates that the syntax element gfv_chroma_key_value[ c ] is present. gfv_chroma_key_present_flag[ c ] equal to 0 indicates that the syntax element gfv_chroma_key_value[ c ] is not present.The variable ChromaKeyDefaultValueFlag is set equal to !( gfv_chroma_key_value_present_flag[ 0 ] | | gfv_chroma_key_value_present_flag[ 1 ] | | gfv_chroma_key_value_present_flag[ 2 ] ).gfv_chroma_key_value[ c ] specifies the chroma key value corresponding to the c-th colour component as follows:- If ChromaKeyDefaultValueFlag is equal to 1, the variables GfvChromaKeyValue[ c ] are specified as follows:- GfvChromaKeyValue[ 0 ] is set equal to 50.- GfvChromaKeyValue[ 1 ] is set equal to 220.- GfvChromaKeyValue[ 2 ] is set equal to 100.- Otherwise, ChromaKeyDefaultValueFlag is equal to 0, the following applies:- If gfv_chroma_key_value_present_flag[ c ] is equal to 1, GfvChromaKeyValue[ c ] is set equal to the value of gfv_chroma_key_value[ c ].- Otherwise, gfv_chroma_key_value_present_flag[ c ] is equal to 0, GfvChromaKeyValue[ c ] is not specified by this Specification.gfv_chroma_key_thr_present_flag[ i ] equal to 1 indicates that the syntax element gfv_chroma_thr_value[ i ] is present. gfv_chroma_key_thr_present_flag[ i ] equal to 0 indicates gfv_chroma_key_thr_value[ i ] is not present.gfv_chroma_key_thr_value[ i ], when present, specifies the i-th chroma key threshold value. The value of gfv_chroma_key_thr_value[ i ] shall be in the range of 0 to 255, inclusive. When not present, the value of gfv_chroma_key_thr_value[ i ] is inferred as follows:If i is equal to 0, gfv_chroma_key_thr_value[ 0 ] is set equal to 48.Otherwise, i is equal to 1, gfv_chroma_key_thr_value[ 1 ] is set equal to 75.NOTE 3 - The syntax elements gfv_chroma_key_value_present[ c ], gfv_chroma_key_value[ c ], and gfv_chroma_key_thr_value[ i ] could be used to determine a transparency indicator for fusion of the generated face picture and background picture. For example, a transparency indicator, denoted as alpha[ x ][ y ] , for picture sample value, denoted as I[ c ][ x ][ y ] with bitDepth[ c ], where bitDepth[ 0 ] is equal to BitDepth. Y , bitDepth[ 1 ] and bitDepth[ 2 ] are equal to BitDepth C, and GfvChromaKey[ c ] values for sample coordinates x, y, and colour components c could be determined as follows:d[ x ][ y ]= 0for( c = 0; c < 3; c++ )if( gfv_chroma_key_value_present[ c ] | | ChromaKeyDefaultValueFlag )d[ x ][ y ] += ( I[ c ][ x ][ y ] / ( 1 << (bitDepth[ c ] - 8) ) - GfvChromaKeyValue[ c ] ) 2if( ( d[ x ][ y ] < gfv_chroma_key_thr_value[ 0 ] )alpha[ x ][ y ] = 0else if ( ( d[ x ][ y ] > gfv_chroma_key_thr_value[ 1 ] )alpha[ x ][ y ] = 1elsealpha[ x ][ y ] = ( d[ x ][ y ] - gfv_chroma_key_thr_value[ 0 ] ) / ( gfv_chroma_key_thr_value[ 1 ] - gfv_chroma_key_thr_value[ 0 ] )A value of alpha[ x ][ y ] equal to 0 could indicate transparency. A value of alpha[ x ][ y ] equal to 1 could indicate opacity. Intermediate values of alpha[ x ][ y ] could indicate semitransparency.gfv_drive_pic_fusion_flag, when present, equal to 1 indicates that the current decoded picture, which corresponds to a driving picture that may be used for fusion, may be input to GenerativeNN( ). gfv_drive_pic_fusion_flag equal to 0 indicates that the current decoded picture should not be input to GenerativeNN( ).NOTE 4 - A gfv_drive_pic_fusion_flag value of 1 can be used, for example, to indicate that the current decoded picture can be used to improve face details or handle background changes.NOTE 5 - When gfv_base_pic_flag is equal to 0 and gfv_drive_pic_fusion_flag is equal to 1, the GFV process takes three inputs: the base picture, features from keypoints and / or matrices carried in the GFV SEI message, and the current decoded picture that is a fusion picture, and outputs a picture that is generated by the GenerativeNN( ).NOTE 6 - When gfv_base_pic_flag is equal to 0 and gfv_drive_pic_fusion_flag is equal to 0, the GFV process takes twoinputs: the base picture and features from keypoints and / or matrices carried in the GFV SEI message, and outputs a picture that is generated by the GenerativeNN( ).NOTE 7 - When gfv_base_pic_flag is equal to 1, the GFV process directly outputs the cropped decoded picture.When a GFV SEI message has gfv_base_pic_flag equal to 0 and gfv_drive_pic_fusion_flag equal to 0, the GFV SEI message pertains to the current decoded picture only.When a GFV SEI message with a particular gfv_id value has gfv_base_pic_flag equal to 0 and gfv_drive_pic_fusion_flag equal to 1, the fusion picture for that particular gfv_id value, which is the current cropped decoded picture, remains valid for the current decoded picture and all subsequent decoded pictures of the current layer, in output order, until the end of the current CLVS or up to but excluding the decoded picture that is within the current CLVS, follows the current decoded picture in output order, and is associated with a GFV SEI message having that particular gfv_id value, whichever is earlier.When a GFV SEI message gfvSeiA with a particular gfv_id value has gfv_cnt greater than 0 and a GFV SEI message gfvSeiB with the same gfv_id value in the same picture unit has gfv_base_pic_flag equal to 1 (i.e., the current decoded picture is a base picture), the GFV SEI message gfvSeiA shall have gfv_drive_pic_fusion_flag equal to 0.gfv_low_confidence_face_parameter_flagequal to 1 indicates the facial parameters have been derived with low confidence. gfv_low_confidence_face_parameter_flag equal to 0 indicates the confidence information of the facial parameters is not specified.gfv_coordinate_present_flagequal to 1 indicates that coordinate information of keypoints is present. gfv_coordinate_present_flag equal to 0 indicates that coordinate information of keypoints is not present.It is a requirement of bitstream conformance that when gfv_matrix_type_idx[ i ] for any i from 0 to gfv_num_matrix_types_minus1 is equal to 0 or 1, the value of gfv_coordinate_present_flag shall be equal to 1.gfv_kps_pred_flagequal to 1 indicates that the syntax elements gfv_coordinate_dx_abs[ i ] ,gfv_coordinate_dy_abs[ i ], and gfv_coordinate_dz_abs[ i ] are present and the syntax elements gfv_coordinate_dx_sign_flag[ i ], gfv_coordinate_dy_sign_flag[ i ] and gfv_coordinate_dz_sign_flag[ i ] may be present.gfv_kps_pred_flag equal to 0 indicates that the syntax elements gfv_coordinate_x_abs[ i ], gfv_coordinate_y_abs[ i ], and gfv_coordinate_z_abs[ i ] are present and the syntax elements gfv_coordinate_x_sign_flag[ i ], gfv_coordinate_y_sign_flag[ i ] and gfv_coordinate_z_sign_flag[ i ] may be present.When gfv_coordinate_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_kps_pred_flag is equal to 1, there shall be a previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1 in the current CLVS.gfv_coordinate_precision_factor_minus1plus 1 indicates the precision of key point coordinates signgalled in the SEI message. The value of gfv_coordinate_precision_factor_minus1 shall be in the range of 0 to 31, inclusive.When gfv_coordinate_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_kps_pred_flag is equal to 1, the value of gfv_coordinate_precision_factor_minus1 is inferred to be equal to the gfv_coordinate_precision_factor_minus1 of the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_num_kps_minus1plus 1 indicates the number of keypoints. The value of gfv_num_kps_minus1 shall be in the range of 0 to 2. 10- 1, inclusive. When gfv_coordinate_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_kps_pred_flag is equal to 1, the value of gfv_num_kps_minus1 is inferred to be equal to the gfv_num_kps_minus1 of the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_coordinate_z_present_flagequal to 1 indicates that z-axis coordinate information of the keypoints is present. gfv_coordinate_z_present_flagequal to 0 indicates that the z-axis coordinate information of the keypoints is not present. When gfv_coordinate_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_kps_pred_flag is equal to 1, the value of coordinate_z_present_flag is inferred to be equal to the coordinate_z_present_flag of the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_coordinate_z_max_value_minus1plus 1 indicates the maximum absolute value of z-axis coordinates of keypoints. The value of gfv_coordinate_z_max_value_minus1 shall be in the range of 0 to 2. 16- 1, inclusive. When gfv_coordinate_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_kps_pred_flag is equal to 1, the value of gfv_coordinate_z_max_value_minus1 is inferred to be equal to the gfv_coordinate_z_max_value_minus1, when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_coordinate_x_abs[ i ] is used to derive the x-axis coordinate of the i-th keypoint. The value of gfv_coordinate_x_abs[ i ] shall be in the range of 0 to 2gfv_coordinate_precision_factor_minus1 + 1, inclusivegfv_coordinate_x_sign_flag[ i ] specifies the sign of the x-axis coordinate of the i-th keypoint. When gfv_coordinate_x_sign_flag[ i ] is not present, it is inferred to be equal to 0.gfv_coordinate_y_abs[ i ] is used to derive y-axis coordinate of i-th keypoint. The value of gfv_coordinate_y_abs[ i ] shall be in the range of 0 to 2gfv_coordinate_precision_factor_minus1 + 1,inclusive.gfv_coordinate_y_sign_flag[ i ] specifies the sign of the y-axis coordinate of the i-th keypoint. When gfv_coordinate_y_sign_flag[i] is not present, it is inferred to be equal to 0.gfv_coordinate_z_abs[ i ] is used to derive z-axis coordinate of the i-th keypoint. The value of gfv_coordinate_z_abs[ i ] shall be in the range of 0 to 2gfv_coordinate_precision_factor_minus1 + 1, inclusive.gfv_coordinate_z_sign_flag[ i ] specifies the sign of the z-axis coordinate of the i-th key point. When gfv_coordinate_z_sign_flag[ i ] is not present, it is inferred to be equal to 0.gfv_coordinate_dx_abs[ i ] specifies a difference value that is used to derive x-axis coordinate of the i-th keypoint. The value of gfv_coordinate_dx_abs[ i ] shall be in the range of 0 to 2gfv_coordinate_precision_factor_minus1 + 2,inclusive.gfv_coordinate_dx_sign_flag[ i ] specifies the sign of the difference value of the x-axis coordinate of the i-th keypoint. When gfv_coordinate_dx_sign_flag[ i ] is not present, it is inferred to be equal to 0.gfv_coordinate_dy_abs[ i ] specifies a difference value that is used to derive y-axis coordinate of the i-th keypoint. The value of gfv_coordinate_dy_abs[ i ] shall be in the range of 0 to 2gfv_coordinate_precision_factor_minus1 + 2, inclusive.gfv_coordinate_dy_sign_flag[ i ] specifies the sign of the difference value of the y-axis coordinate of the i-th keypoint. When gfv_coordinate_yd_sign_flag[i] is not present, it is inferred to be equal to 0.gfv_coordinate_dz_abs[ i ] specifies a difference value that is used to derive z-axis coordinate of the i-th keypoint. The value of gfv_coordinate_dz_abs[ i ] shall be in the range of 0 to 2gfv_coordinate_precision_factor_minus1 + 2,inclusive.gfv_coordinate_dz_sign_flag[ i ] specifies the sign of the difference value of the z-axis coordinate of the i-th key point. When gfv_coordinate_dz_sign_flag[ i ] is not present, it is inferred to be equal to 0.If gfv_coordinate_z_max_value_minus1 is present,the variable CroppedDepth is set equal to gfv_coordinate_z_max_value_minus1 + 1. Otherwise, CroppedDepth is set equal to 0.When gfv_kps_pred_flag is equal to 1, the variables coordinateDeltaX[ i ], coordinateDeltaY[ i ] and coordinateDeltaZ[ i ] indicating the delta x-axis coordinate, delta y-axis coordinate and delta z-axis coordinate of the i-th keypoint, respectively,are derived as follows:coordinateDeltaX[ i ] = ( 1 - 2 * gfv_coordinate_dx_sign_flag[ i ] ) * gfv_coordinate_dx_abs[ i ] / ( 1 << ( gfv_coordinate_precision_factor_minus1 + 1 ) )coordinateDeltaY[ i ] = ( 1 - 2 * gfv_coordinate_dy_sign_flag[ i ] ) * gfv_coordinate_dy_abs[ i ] / ( 1 << ( gfv_coordinate_precision_factor_minus1 + 1 ) )if( gfv_coordinate_z_present_flag )coordinateDeltaZ[ i ] = ( 1 - 2 * gfv_coordinate_dz_sign_flag[ i ] ) * gfv_coordinate_dz_abs[ i ] / ( 1 << ( gfv_coordinate_precision_factor_minus1 + 1 ) )The variables coordinateX[ i ], coordinateY[ i ], and, when gfv_coordinate_z_present_flag is equal to 1, coordinateZ[ i ] indicating the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th keypoint, respectively, are derived as follows:If gfv_kps_pred_flag is equal to 0,the following applies:coordinateX[ i ] = ( 1 - 2 * gfv_coordinate_x_sign_flag[ i ] ) * gfv_coordinate_x_abs[ i ] / ( 1 << ( gfv_coordinate_precision_factor_minus1 + 1 ) )coordinateY[ i ] = ( 1 - 2 * gfv_coordinate_y_sign_flag[ i ] ) * gfv_coordinate_y_abs[ i ] / ( 1 << ( gfv_coordinate_precision_factor_minus1 + 1 ) )if (gfv_coordinate_z_present_flag )coordinateZ[ i ] = ( 1 - 2 * gfv_coordinate_z_sign_flag[ i ] ) * gfv_coordinate_z_abs[ i ] / ( 1 << ( gfv_coordinate_precision_factor_minus1 + 1 ) )Otherwise (gfv_kps_pred_flag is equal to 1),the following applies:if( gfv_base_pic_flag ) {coordinateX[ i ] = (( i > 0 ) ? coordinateX[ i - 1 ] : 0 ) + coordinateDeltaX[ i ]coordinateY[ i ] = (( i > 0 ) ? coordinateY[ i - 1 ] : 0 ) + coordinateDeltaY[ i ]if (gfv_coordinate_z_present_flag )coordinateZ[ i ] = (( i > 0 ) ? coordinateZ[ i - 1 ] : 0 ) + coordinateDeltaZ[ i ]} else if( gfv_cnt = = 0 ) {coordinateX[ i ] = BaseKpCoordinateX[ i ] + coordinateDeltaX[ i ]coordinateY[ i ] = BaseKpCoordinateY[ i ] + coordinateDeltaY[ i ]if (gfv_coordinate_z_present_flag )coordinateZ[ i ] = BaseKpCoordinateZ[ i ] + coordinateDeltaZ[ i ]} else {coordinateX[ i ] = PrevKpCoordinateX[ i ] + coordinateDeltaX[ i ]coordinateY[ i ] = PrevKpCoordinateY[ i ] + coordinateDeltaY[ i ]coordinateZ[ i ] = PrevKpCoordinateZ[ i ] + coordinateDeltaZ[ i ]}The following applies for derivation of the variables BaseKpCoordinateX[ i ], BaseKpCoordinateY[ i ], BaseKpCoordinateZ[ i ], PrevKpCoordinateX[ i ], PrevKpCoordinateY[ i ],and PrevKpCoordinateZ[ i ]:if( gfv_base_pic_flag ) {PrevKpCoordinateX[ i ] = BaseKpCoordinateX[ i ] = coordinateX[ i ]PrevKpCoordinateY[ i ] = BaseKpCoordinateY[ i ] = coordinateY[ i ]if (gfv_coordinate_z_present_flag )PrevKpCoordinateZ[ i ] = BaseKpCoordinateZ[ i ] = coordinateZ[ i ]} else {PrevKpCoordinateX[ i ] = coordinateX[ i ]PrevKpCoordinateY[ i ] = coordinateY[ i ]PrevKpCoordinateZ[ i ] = coordinateZ[ i ]}gfv_matrix_present_flagequal to 1 indicates that matrix parameters are present. gfv_matrix_present_flag equal to 0 indicates that matrix parameters are not present. When gfv_coordinate_present_flag is equal to 0,gfv_matrix_present_flag shall be equal to 1.gfv_matrix_pred_flagequal to 0 indicates that the syntax elements gfv_matrix_element_int[ i ][ j ][ k ][ m ] and gfv_matrix_element_dec[ i ][ j ][ k ][ m ] are present and the syntax element gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ] may be present. gfv_matrix_pred_flag equal to 1 indicates that the syntax elements gfv_matrix_delta_element_int[ i ][ j ][ k ][ m ] and gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ] are present and the syntax element gfv_matrix_delta_element_sign_flag [ i ][ j ][ k ][ m ] may be present. When gfv_matrix_pred_flag is not present, it is inferred to be 0.When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_matrix_pred_flag is equal to 1,there shall be a previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1 in the current CLVS.gfv_matrix_element_precision_factor_minus1plus 1 indicates the precision of matrix elements signalled in the SEI message. The value of gfv_matrix_element_precision_factor_minus1 shall be in the range of 0 to 31, inclusive. When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_matrix_pred_flag is equal to 1, the value of gfv_matrix_element_precision_factor_minus1 is inferred to be equal to the gfv_matrix_element_precision_factor_minus1 of the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_num_matrix_types_minus1plus 1indicates the number of matrix types signalled in the SEI message. The value of gfv_num_matrix_types_minus1 shall be in the range of 0 to 2, 6- 1, inclusive. It is a requirement of bitstream conformance that when gfv_matrix_pred_flag is equal to 1 and gfv_base_pic_flag is equal to 0, the value of gfv_num_matrix_types_minus1 shall be equal to the value of gfv_num_matrix_types_minus1 in each of the preceding GFV SEI message in decoding order in the current CLVS which has the same gfv_id value as the gfv_id value in the current SEI and has gfv_base_pic_flag equal to 1. When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, and gfv_matrix_pred_flag is equal to 1, the value of gfv_num_matrix_types_minus1 is inferred to be equal to the gfv_num_matrix_types_minus1of the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_matrix_type_idx[ i ]indicates the index of the i-th matrix type as specified in Table A. The value of gfv_matrix_type_idx[ i ] shall be in the range of 0 to 63, inclusive.In bitstreams conforming to this version of this Specification, the value of gfv_matrix_type_idx[ i ] shall be in the range of 0 to 31, inclusive. Decoders conforming to this version of this Specification shall allow gfv_matrix_type_idx[ i ] to be greater than 31 to appear in the bitstream and the decoder shall ignore all information for the i-th type of matrix for which gfv_matrix_type_idx[ i ] is greater than 31.Table A - Specification of gfv_matrix_type_idx[ i ]. NOTE 8 - The undefined matrxi type is used to represent the matrxi type rather than affine translation matrix, covariance matrix, rotation matrix, translation matrix and compact feature matrix. It can be used by the user to extend the matrix type.gfv_num_matrices_equal_to_num_kps_flag[ i ] equal to 1 indicates that the number of matrices of the i-th matrix type is equal to gfv_num_kps_minus1 + 1. gfv_num_matrices_equal_to_num_kps_flag[ i ] equal to 0 indicates the number of matrices of the i-th matrix type is not equal to gfv_num_kps_minus1 + 1.If gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, gfv_matrix_pred_flag is equal to 1, gfv_matrix_type_idx[ i ] is equal to 0 or 1, and gfv_coordinate_present_flag is equal to 1, the value of gfv_num_matrices_equal_to_num_kps_flag[ i ] is inferred to be equal to the gfv_num_matrices_equal_to_num_kps_flag[ i ], when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1. Otherwise, when gfv_num_matrices_equal_to_num_kps_flag[ i ] is not present, its vlaue is inferred to be equal to 0.gfv_num_matrices_info[ i ] provides information to derive the number of the matrices of the i-th matrix type. The value of gfv_num_matrices_info[ i ] shall be in the range of 0 to 2. 10- 1, inclusive. When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, gfv_matrix_pred_flag is equal to 1, gfv_matrix_type_idx[ i ] is equal to 0 or 1, and either gfv_coordinate_present_flag is equal to 0 or gfv_num_matrix_equal_to_num_kps_flag[ i ] is equal to 0, the value of gfv_num_matrices_info[ i ] is inferred to be equal to the gfv_num_matrices_info[ i ], when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_matrix_width_minus1[ i ] plus 1 indicates the width of the matrix of the i-th matrix type. The value of gfv_matrix_width_minus1[ i ] shall be in the range of 0 to 2 10- 1, inclusive. When gfv_matrix_present_flag is equal to 1, gfv_matrix_pred_flag is equal to 0, gfv_matrix_pred_flag is equal to 1, and gfv_matrix_type_idx[ i ] is equal to 2 or 3 or is greater than or equal to 7, the value of gfv_matrix_width_minus1[ i ] is inferred to be equal to the gfv_matrix_width_minus1[ i ], when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_matrix_height_minus1[ i ] plus 1 indicates the height of the matrix of the i-th matrix type. The value of gfv_matrix_height_minus1[ i ] shall be in the range of 0 to 2 10- 1, inclusive. When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, gfv_matrix_pred_flag is equal to 1, and gfv_matrix_type_idx[ i ] is equal to 2 or 3 or is greater than or equal to 7, the value of gfv_matrix_height_minus1[ i ] is inferred to be equal to the gfv_matrix_height_minus1[ i ], when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.gfv_matrix_for_3D_space_flag[ i ] equal to 1 indicates the matrix of the i-th matrix type is a matrix defined in three-dimensional space. gfv_matrix_for_3D_space_flag[ i ] equal to 0 indicates the matrix of the i-th matrix type is a matrix defined in two-dimensional space.When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, gfv_matrix_pred_flag is equal to 1, gfv_matrix_type_idx[ i ] is equal to 4, 5, or 6, and gfv_coordinate_present_flag is equal to 0, the value of gfv_matrix_for_3D_space_flag[ i ] is inferred to be equal to the gfv_matrix_for_3D_space_flag[ i ], when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.When gfv_matrix_width_minus1[ i ] is not present, it is inferred as follows:If gfv_matrix_type_idx[ i ] is equal to 0, 1 or 4, and one of coordinate_z_present_flag and gfv_matrix_for_3D_space_flag[ i ] is present and equal to 1, gfv_matrix_width_minus1[i] is inferred to be equal to 2.Otherwise, if gfv_matrix_type_idx[ i ] is equal to 0, 1 or 4, and one of coordinate_z_present_flag and gfv_matrix_for_3D_space_flag[ i ] is present and equal to 0, gfv_matrix_width_minus1[ i ] is inferred to be equal to 1.Otherwise (gfv_matrix_type_idx[ i ] is equal to 5 or 6), gfv_matrix_width_minus1[ i ] is inferred to be equal to 0.When gfv_matrix_height_minus1[ i ] is not present, it is inferred as follows:If gfv_matrix_type_idx is equal to 0, 1, 4, 5 or 6, and one of gfv_coordinate_z_present_flag and gfv_matrix_for_3D_space_flag[ i ] is present and equal to 1, gfv_matrix_height_minus1[ i ] is inferred to be equal to 2.Otherwise (gfv_matrix_type_idx is equal to 0, 1, 4, 5 or 6, and one of gfv_coordinate_z_present_flag and gfv_matrix_for_3D_space_flag[ i ] is 0), gfv_matrix_height _minus1[ i ] is inferred to be equal to 1.The variables matrixWidth[ i ] and matrixHeight[ i ] indicating the width and height of the matrix of the i-th matrix type are derived as follows:if( gfv_matrix_pred_flag ) {matrixWidth[ i ] = BaseMatrixWidth[ i ]matrixHeight[ i ] = BaseMatrixHeight[ i ]} else {matrixWidth[ i ] = gfv_matrix_width_minus1[ i ] + 1matrixHeight[ i ] = gfv_matrix_height_minus1[ i ] + 1}if( gfv_base_pic_flag ) {BaseMatrixWidth[ i ] = matrixWidth[ i ]BaseMatrixHeight[ i ] = matrixHeight[ i ]}gfv_num_matrices_minus1[ i ] plus 1 indicates the number of matrices of the i-th matrix type. The value of gfv_num_matrices_minus1[ i ] shall be in the range of 0 to 2. 10- 1, inclusive. When gfv_matrix_present_flag is equal to 1, gfv_base_pic_flag is equal to 0, gfv_matrix_pred_flag is equal to 1, and gfv_matrix_type_idx[ i ] is greater than or equal to 7, the value of gfv_num_matrices_minus1[ i ] is inferred to be equal to the gfv_num_matrices_minus1[ i ], when present, in the previous GFV SEI message in decoding order with the same gfv_id as the current GFV SEI message and gfv_base_pic_flag equal to 1.The variable numMatrices[ i ] indicating the number of the matrices of the i-th matrix type is derived as follows:if( gfv_matrix_pred_flag )numMatrices[ i ] = BaseNumMatrices[ i ]else if( gfv_matrix_type_idx[ i ] = = 0 | | gfv_matrix_type_idx[ i ] = = 1 ) {if( gfv_coordinate_present_flag )numMatrices[ i ] = gfv_num_matrices_equal_to_num_kps_flag[ i ] ? gfv_num_kps_minus1 + 1:( gfv_num_matrices_info[ i ] < gfv_num_kps_minus1 ? gfv_num_matrices_info [ i ] + 1:gfv_num_matrices_info [ i ] + 2 )elsenumMatrices[ i ] = gfv_num_matrices_info[ i ] + 1}else if( gfv_matrix_type_idx[ i ] >= 2 && gfv_matrix_type_idx[ i ] < 7 )numMatrices[ i ] = 1elsenumMatrices[ i ] = gfv_num_matrices_minus1[ i ] + 1if( gfv_base_pic_flag )BaseNumMatrices[ i ] = numMatrices[ i ]It is a requirement of bitstream conformance that when gfv_matrix_pred_flag is equal to 1 and gfv_base_pic_flag is equal to 0, the values of numMatrices[ i ], matrixWidth[ i ], and matrixHeight[ i ] for i in the range of 0 to gfv_num_matrix_types_minus1, inclusive shall be respectively equal to the values of numMatrices[ i ], matrixWidth[ i ], and matrixHeight[ i ] for i in the range of 0 to gfv_num_matrix_types_minus1, inclusive in each of the preceding GFV SEI message in decoding order in the current CLVS which has the same gfv_id value as the gfv_id value in the current SEI and has gfv_base_pic_flag equal to 1.gfv_matrix_element_int[ i ][ j ][ k ][ m ] indicates the integer part of the value of the matrix element at position (m, k) of the j-th matrix of the i-th matrixtype. The value of gfv_matrix_element_int[ i ][ j ][ k ][ m ] shall be in the range of 0 to 2 32 - 2, inclusive.gfv_matrix_element_dec[ i ][ j ][ k ][ m ] indicates the decimal part of the value of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type. The length of gfv_matrix_element_dec[ i ][ j ][ k ][ m ] is gfv_matrix_element_precision_factor_minus1 + 1 bits.gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ] indicates the sign of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type. When gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ]is not present, it is inferred to be equal to 0.gfv_matrix_delta_element_int[ i ][ j ][ k ][ m ] indicates the integer part of the difference value of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type. The value of gfv_matrix_delta_element_int[ i ][ j ][ k ][ m ] shall be in the range of 0 to 2 32- 2, inclusive.gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ] indicates the decimal part of the difference value of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type. The value of gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ] shall be in the range of 0 to 2 32- 2, inclusive.gfv_matrix_delta_element_sign_flag[ i ][ j ][ k ][ m ] indicates the sign of the difference value of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type. When gfv_matrix_delta_element_sign_flag[ i ][ j ][ k ][ m ]is not present, it is inferred to be equal to 0.When gfv_matrix_pred_flag is equal to 1, the variable matrixElementDeltaVal[ i ][ j][ k ][ m ] representing the difference value of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type is derived as follows:matrixElementDeltaVal[ i][ j ][ k ][ m ] = (1-2*gfv_matrix_delta_element_sign_flag[ i ][ j ][ k ][ m ]) * (gfv_matrix_delta_element_int[ i ][ j ][ k ][ m ]+(gfv_matrix_delta_element_dec[ i ][ j ][ k ][ m ] / (1<<gfv_matrix_element_precision_factor_minus1+1))The variable matrixElementVal[ i ][ j][ k ][ m ] representing the value of the matrix element at position (m, k) of the j-th matrix of the i-th matrix type is derived as follows:If gfv_matrix_pred_flagis equal to 0, the following applies:matrixElementVal[ i][ j ][ k ][ m ] = (1-2*gfv_matrix_element_sign_flag[ i ][ j ][ k ][ m ]) * (gfv_matrix_element_int[ i ][ j ][ k ][ m ] + (gfv_matrix_element_dec[ i ][ j ][ k ][ m ] / (1<< gfv_matrix_element_precision_factor_minus1+1))if( gfv_base_pic_flag )BaseMatrixElementVal[ i][ j ][ k ][ m ] = matrixElementVal[ i][ j ][ k ][ m ]Otherwise (gfv_matrix_pred_flag is equal to 1), the following applies:if( gfv_cnt = = 0 )matrixElementVal[ i][ j ][ k ][ m ] = BaseMatrixElementVal[ i][ j ][ k ][ m ] +matrixElementDeltaVal[ i][ j ][ k ][ m ]elsematrixElementVal[ i][ j ][ k ][ m ] = PrevMatrixElementVal[ i][ j ][ k ][ m ] +matrixElementDeltaVal[ i][ j ][ k ][ m ]The following applies:if( gfv_base_pic_flag )PrevMatrixElementVal[ i][ j ][ k ][ m ] = BaseMatrixElementVal[ i][ j ][ k ][ m ] =matrixElementVal[ i][ j ][ k ][ m ]elsePrevMatrixElementVal[ i][ j ][ k ][ m ] = matrixElementVal[ i][ j ][ k ][ m ]For a particular gfv_id value, the followingprocess is used in increasing order of gfv_cnt to generate a video picture per each GFV SEI message that has gfv_base_pic_flag equal to 0 and a unique value of gfv_cnt within a picture unit:DeriveSigParam( )TranslatorNN( sigKeyPoint, sigMatrix )DeriveInputTensors( )if( gfv_base_pic_flag = = 0 && gfv_drive_pic_fusion_flag = = 0 ) {if( ChromaFormatIdc = = 0 )GenerativeNN( inputBaseY, inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint,inputDriveMatrix, CroppedWidth, CroppedHeight, CroppedDepth )elseGenerativeNN( inputBaseY, inputBaseCb, inputBaseCr, inputBaseKeyPoint, inputBaseMatrix,inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight, CroppedDepth )} else if( gfv_base_pic_flag = = 0 && gfv_drive_pic_fusion_flag = = 1 ) {if( ChromaFormatIdc = = 0 )GenerativeNN( inputBaseY, inputDriveY, inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint,inputDriveMatrix, CroppedWidth, CroppedHeight, CroppedDepth )elseGenerativeNN( inputBaseY, inputBaseCb, inputBaseCr,inputDriveY, inputDriveCb, inputDriveCr,inputBaseKeyPoint, inputBaseMatrix,, inputDriveKeyPoint, inputDriveMatrix,CroppedWidth, CroppedHeight, CroppedDepth )}StoreOutputTensors( )The process DeriveSigParam( ) for deriving the inputs of TranslatorNN( ) is specified as follows:The keypoint coordinate array sigKeyPoint and the matrix sigMatrix are derived as follows:if( gfv_coordinate_present_flag )for( i = 0; i <= gfv_num_kps_minus1; i++ ) {sigKeyPoint[ i ][ 0 ] = coordinateX[ i ]sigKeyPoint[ i ][ 1 ] = coordinateY[ i ]if( gfv_coordinate_z_present_flag )sigKeyPoint[ i ][ 2 ] = coordinateZ[ i ]}elsefor( i = 0; i <= gfv_num_kps_minus1; i++ ) {sigKeyPoint[ i ][ 0 ] = 0sigKeyPoint[ i ][ 1 ] = 0if ( gfv_coordinate_z_present_flag )sigKeyPoint[ i ][ 2 ] = 0}if( gfv_matrix_present_flag )for ( i = 0; i <= gfv_num_matrix_types_minus1; i++ )for ( j = 0; j < numMatrices[ i ]; j++ )for( k = 0; k < matrixHeight [ i ]; k++ )for( l = 0;l < matrixWidth [ i ]; l++)sigMatrix[ i ][ j ][ k ][ l ] =matrixElementVal[ i ][ j][ k][ l ]elsefor( i = 0; i <= gfv_num_matrix_types_minus1; i++ )for ( j = 0; j < numMatrices[ i ]; j++ )for( k = 0; k < matrixHeight [ i ]; k++ )for( l = 0;l < matrixWidth [ i ]; l++)sigMatrix[ i ][ j ][ k ][ l ] = 0TranslatorNN( ) is a process to translate the various formats of the facial parameters carried in the SEI message to the fixed format of the facial parameters to be input to the generative network to generate the output pictureInputs to TranslatorNN( ) are:sigKeyPoint and sigMatrixOutputs of TranslatorNN( ) are:convKeyPoint and convNumKeyPointconvMatrix and convNumMatrix, convMatrixWidth, convMatrixHeightThe process DeriveInputTensors( ) for deriving the inputs of GenerativeNN( ) is specified as follows:When gfv_base_pic_flag is equal to 1, the BasePicture input tensor inputBaseY, inputBaseCb and inputBaseCr are derived as follows:for( x = 0; x < CroppedWidth; x++ )for ( y = 0; y < CroppedHeight; y++ )inputBaseY[ x ][ y ] = InpY(baseCroppedYPic[ x ][ y ] )if( ChromaFormatIdc != 0 )for( x = 0; x < CroppedWidth / SubWidthC; x++ )for ( y = 0; y < CroppedHeight / SubHeightC; y++ ) {inputBaseCb[ x][ y ] = InpC( baseCroppedCbPic[ x][ y ] )inputBaseCr[ x][ y ] = InpC( baseCroppedCrPic[ x ][ y ] )}When gfv_base_pic_flag is equal to 0 and gfv_drive_pic_fusion_flag is equal to 1, the DrivePicture luma sample array inputDriveY, inputDriveCb and input DriveCr are derived as follows:for( x = 0; x< CroppedWidth; x++ )for ( y = 0; y< CroppedHeight; y++ )inputDriveY[ x ][ y ] = InpY( driveCroppedYPic[ x ][ y ] )if( ChromaFormatIdc != 0 )for( x = 0; x< CroppedWidth / SubWidthC; x++ )for( y = 0; y < CroppedHeight / SubHeightC; y++ ) {InputDriveCb[ x][ y ] = InpC( driveCroppedCbPic[ x][ y ] )InputDriveCr[ x][ y ] = InpC( driveCroppedCrPic[ x ][ y ] )}When gfv_base_pic_flag is equal to 0, the keypoint coordinate array inputDriveKeyPoint and the matrix inputDriveMatrix for the current picture are derived as follows:for( i = 0;i < = convNumKeyPoint; i++ ) {inputDriveKeyPoint[ i ][ 0 ] = convKeyPoint[ i ][ 0 ]inputDriveKeyPoint [ i ][ 1 ] = convKeyPoint[ i ][ 1 ]inputDriveKeyPoint [ i ][ 2 ] = convKeyPoint[ i ][ 2 ]}for( j = 0; j < convNumMatrix; j++ )for( k = 0; k < convMatrixHeight; k++ )for( m = 0; m < convMatrixWidth; m++ )inputDriveMatrix[ j ][ k ][ m ] = convMatrix [ j ][ k ][ m ]When gfv_base_pic_flag is equal to 1, the keypoint coordinate array inputBaseKeyPoint and the matrix inputBaseMatrix for the base picture are derived as follows:for( i = 0; i <= convNumKeyPoint; i++ ) {inputBaseKeyPoint[ i ][ 0 ] = convKeyPoint[ i ][ 0 ]inputBaseKeyPoint [ i ][ 1 ] = convKeyPoint[ i ][ 1 ]inputBaseKeyPoint [ i ][ 2 ] = convKeyPoint[ i ][ 2 ]}for( j = 0; j < convNumMatrix; j++ )for( k = 0; k < convMatrixHeight; k++ )for( l = 0; l < convMatrixWidth; l++ )inputBaseMatrix[ j ][ k ][ l ] = convMatrix [ j ][ k ][ l ]Where the functions InpY( ) and InpC( ) are specified as follows:InpY( x ) = x / ( ( 1 << BitDepthY ) - 1 )InpC( x ) = x / ( ( 1 << BitDepth C) - 1 )GenerativeNN( ) is a process to generate the sample values of an output picture corresponding to a driving picture. It is only invoked when gfc_base_pic_flag is equal to 0. Input values to GenerativeNN( ) and output values from GenerativeNN( ) are real numbers.Inputs to GenerativeNN( ) are:When gfv_base_pic_flag is equal to 0, gfv_drive_pic_fusion_flag is equal to 0, and ChromaFormatIdc is equal to 0: inputBaseY, inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight,and CroppedDepth.When gfv_base_pic_flag is equal to 0, gfv_drive_pic_fusion_flag is equal to 0, and ChromaFormatIdc is not equal to 0: inputBaseY, inputBaseCb, inputBaseCr, inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight, and CroppedDepth.When gfv_base_pic_flag is equal to 0, gfv_drive_pic_fusion_flag is equal to 1, gfv_chroma_key_info_present_flag is equal to 0, and ChromaFormatIdc is equal to 0: inputBaseY,inputDriveY, inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight, and CroppedDepth.When gfv_base_pic_flag is equal to 0, gfv_drive_pic_fusion_flag is equal to 1, gfv_chroma_key_info_present_flag is equal to 0, and ChromaFormatIdc is not equal to 0: inputBaseY, inputBaseCb, inputBaseCr, inputDriveY, inputDriveCb, inputDriveCr , inputBaseKeyPoint, inputBaseMatrix,, inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight, and CroppedDepth.When gfv_base_pic_flag is equal to 0, gfv_drive_pic_fusion_flag is equal to 1, gfv_chroma_key_info_present_flag is equal to 1, and ChromaFormatIdc is equal to 0: inputBaseY, inputDriveY, inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight, CroppedDepth, gfv_chroma_key_thr_value[ 0 ], gfv_chroma_key_thr_value[ 1 ], and GfvChromaKeyValue[ 0 ].When gfv_base_pic_flag is equal to 0, gfv_drive_pic_fusion_flag is equal to 1,gfv_chroma_key_info_present_flag is equal to 0, and ChromaFormatIdc is not equal to 0: inputBaseY, inputBaseCb, inputBaseCr, inputDriveY, inputDriveCb, inputDriveCr , inputBaseKeyPoint, inputBaseMatrix, inputDriveKeyPoint, inputDriveMatrix, CroppedWidth, CroppedHeight, CroppedDepth, gfv_chroma_key_thr_value[ 0 ], gfv_chroma_key_thr_value[ 1 ], and, when specified, GfvChromaKeyValue[ 0 ], GfvChromaKeyValue[ 1 ], and GfvChromaKeyValue[ 2 ].Outputs of GenerativeNN( ) are:A luma sample array genYWhen ChromaFormatIdc is not equal to 0, two chroma sample arrays genCb and genCr.The process StoreOutputTensors( ) for deriving the output is specified as follows:When gfv_base_pic_flag is equal to 0, the output sample array outYPic[ x ][ y ], outCbPic[ x ][ y ],and outCrPic[ x ][ y ] are derived as follows:for( x = 0; x < CroppedWidth; x++ )for( y = 0; y < CroppedHeight; y++ )outputYPic[ x ][ y ] = OutY( genY[ x ][ y ] )if( ChromaFormatIdc != 0 )for( x = 0; x < CroppedWidth / SubWidthC; x++ )for( y = 0; y < CroppedHeight / SubHeightC; y++ ) {outputCbPic[ x ][ y ] = OutC( genCb[ x ][ y ] )outputCrPic[ x][ y ] = OutC( genCr[ x ][ y ] )}When gfv_base_pic_flag is equal to 1, the output sample array outYPic[ x ][ y ], outCbPic[ x ][ y ],and outCrPic[ x ][ y ] are derived as follows (each output picture derived by the process StoreOutputTensors( ) is referred to as a GFV-generated picture):for( x = 0; x< CroppedWidth; x++ )for( y = 0; y< CroppedHeight; y++ )outputYPic[ x ][ y ] = baseCroppedYPic[ x ][ y ]if( ChromaFormatIdc != 0 )for( x = 0; x< CroppedWidth / SubWidthC; x++ )for( y = 0; y< CroppedHeight / SubHeightC; y++ ) {outputCbPic[ x ][ y ] = baseCroppedCbPic[ x ][ y ]outputCrPic[ x][ y ] = baseCroppedCbPic[ x ][ y ]}Where the functions OutY( ) and OutC( ) are specified as follows:OutY( x ) = Clip3( 0, ( 1 << BitDepth, Y ) - 1 , x * ( ( 1 << BitDepth Y ) - 1 )OutC( x ) = Clip3( 0, ( 1 << BitDepth C ) - 1 , x * ( ( 1 << BitDepth C) - 1 )The output order of GFV-generated pictures corresponding to the GFV SEI messages in a picture unit with the same gfv_id value and different gfv_cnt values shall be in increasing order of the gfv_cnt values. For any two pictures picA and picB wherein picA precedes picB in output order, any GFV-generated picture corresponding to a GFV SEI message with a particular gfv_id value and associated with picA shall precede, in output order, any GFV-generated picture corresponding to a GFV SEI message with the particular gfv_id value and associated with picB.

[0102] Generative face video SEI messages have a problem in that unnecessary operations may occur during the implementation process due to redundancy in the existence conditions of matrix information included in the generative face video SEI messages. Here, the matrix information may be a concept including gfv_coordinate_precision_factor_minus1 to gfv_matrix_for_3D_space_flag[i] of Table 1. For example, the matrix information may be signaled when the base picture flag (gfv_base_pic_flag) is 1 or when the matrix prediction flag (gfv_matrix_pred_flag) is 0. In this case, the matrix prediction flag may be signaled when the base picture flag is 0, and when the matrix prediction flag is not signaled (i.e., when the base picture flag is 1), the value of the matrix prediction flag may be considered as 0. Therefore, the case where the reference picture flag is 1 and the case where the matrix prediction flag is 0 may correspond to the same condition, and due to the redundancy of these conditions, unnecessary operations may occur during the implementation process.

[0103] In the embodiments to be described below, we will examine a method for eliminating redundancy in the existence conditions of matrix information included in a generated face video SEI message.

[0104] Example 1

[0105] Table 3 is an example of a generated face video SEI message included in a bitstream.

[0106] generative_face_video ( payloadSize ) {Descriptor...gfv_matrix_present_flagu(1)if( gfv_matrix_present_flag ) {if( !gfv_base_pic_flag )gfv_matrix_pred_flagu(1)if( !gfv_matrix_pred_flag ) {gfv_matrix_element_precision_factor_minus1ue(v)gfv_num_matrix_types_minus1ue(v)for( i = 0; i <= num_matrix_types_minus1; i++ ) {gfv_matrix_type_idx[ i ]u(6)if( gfv_matrix_type_idx[ i ] = = 0 | | gfv_matrix_type_idx[ i ] = = 1 ) {if( gfv_coordinate_present_flag )gfv_num_matrices_equal_to_num_kps_flag[ i ]u(1)if( !gfv_num_matrices_equal_to_num_kps_flag[ i ] )gfv_num_matrices_info[ i ]ue(v)} else if( gfv_matrix_type_idx[ i ] = = 2 | | gfv_matrix_type_idx[ i ] = = 3 | |gfv_matrix_type_idx[ i ] >= 7 ) {if( gfv_matrix_type_idx[ i ] >= 7 )gfv_num_matrices_minus1[ i ]ue(v)gfv_matrix_width_minus1[ i ]ue(v)gfv_matrix_height_minus1[ i ]ue(v)} else if( gfv_matrix_type_idx[ i ] >= 4 && gfv_matrix_type_idx[ i ] <= 6 &&!gfv_coordinate_present_flag )gfv_matrix_for_3D_space_flag[ i ]u(1)}}...

[0107] Generated face video SEI messages may include a matrix present flag (gfv_matrix_present_flag). The matrix present flag may indicate whether a matrix parameter exists. For example, if gfv_matrix_present_flag is 1, it may indicate that a matrix parameter exists, and if gfv_matrix_present_flag is 0, it may indicate that a matrix parameter does not exist. Here, the matrix parameter may be a concept including at least one of gfv_matrix_pred_flag, gfv_matrix_element_precision_factor_minus1, gfv_num_matrix_types_minus1, gfv_matrix_type_idx[i], gfv_num_matrices_equal_to_num_kps_flag[i], gfv_num_matrices_info[i], gfv_num_matrices_minus1[i], gfv_matrix_width_minus1[i], gfv_matrix_height_minus1[i], or gfv_matrix_for_3D_space_flag[i].

[0108] Generated face video SEI messages may include a base picture flag (gfv_base_pic_flag). The base picture flag may indicate whether the currently decoded output picture corresponds to the base picture. For example, if the value of gfv_base_pic_flag is 1, it may indicate that the currently decoded output picture corresponds to the base picture. Additionally, if the value of gfv_base_pic_flag is 0, it may indicate that the currently decoded output picture does not correspond to the base picture or that the current generated face video SEI message does not contain syntax elements for the base picture. If gfv_base_pic_flag is not present in the current generated face video SEI message, the value of gfv_base_pic_flag may be considered 0.

[0109] Generative face video SEI messages may include a matrix prediction flag (gfv_matrix_pred_flag). The matrix prediction flag may indicate whether matrix element values ​​are signaled based on matrix element difference value information. For example, if gfv_matrix_pred_flag is 1, it may indicate that matrix element difference value information exists, and if gfv_matrix_pred_flag is 0, it may indicate that matrix element value information exists. Additionally, if gfv_matrix_pred_flag is 1, it may indicate that a matrix element difference value sign flag (gfv_matrix_delta_element_sign_flag[i][j][k][m]) may exist, and if gfv_matrix_pred_flag is 0, it may indicate that a matrix element value sign flag (gfv_matrix_element_sign_flag[i][j][k][m]) may exist.

[0110] The above matrix element difference value information may include at least one of matrix element integer difference value information (gfv_matrix_delta_element_int[i][j][k][m]), or matrix element fractional difference value information (gfv_matrix_delta_element_dec[i][j][k][m]), and the above matrix element value information may include at least one of matrix element integer value information (gfv_matrix_element_int[i][j][k][m]), or matrix element fractional value information (gfv_matrix_element_dec[i][j][k][m]).

[0111] gfv_matrix_delta_element_int[i][j][k][m] represents the integer part of the difference value of the matrix element at position (m, k) in the j-th matrix of the i-th matrix type. The value of gfv_matrix_delta_element_int[i][j][k][m] ranges from 0 to 2 32 It can have a range up to -2. gfv_matrix_delta_element_dec[i][j][k][m] represents the fractional part of the difference value of the matrix element at position (m, k) in the j-th matrix of the i-th matrix type. The value of gfv_matrix_delta_element_dec[i][j][k][m] ranges from 0 to 2 32 It can have a range up to -2. gfv_matrix_delta_element_sign_flag[i][j][k][m] can represent the sign of the difference value of the matrix element at position (m, k) in the j-th matrix of the i-th matrix type. If gfv_matrix_delta_element_sign_flag[i][j][k][m] does not exist, the value of gfv_matrix_delta_element_sign_flag[i][j][k][m] can be considered 0.

[0112] gfv_matrix_element_int[i][j][k][m] represents the integer part of the value of the matrix element at position (m, k) in the j-th matrix of the i-th matrix type. The value of gfv_matrix_element_int[i][j][k][m] ranges from 0 to 2 32 It can have a range up to -2. gfv_matrix_element_dec[i][j][k][m] can represent the fractional part of the value of the matrix element corresponding to the (m, k) position in the j-th matrix of the i-th matrix type. The bit length of gfv_matrix_element_dec[i][j][k][m] can be the value of gfv_matrix_element_precision_factor_minus1 plus 1. gfv_matrix_element_sign_flag[i][j][k][m] can represent the sign of the matrix element corresponding to the (m, k) position in the j-th matrix of the i-th matrix type. If gfv_matrix_element_sign_flag[i][j][k][m] does not exist, the value of gfv_matrix_element_sign_flag[i][j][k][m] can be considered 0.

[0113] gfv_matrix_pred_flag can be signaled based on gfv_matrix_present_flag and gfv_base_pic_flag. For example, gfv_matrix_pred_flag can be signaled when gfv_matrix_present_flag is 1 and gfv_base_pic_flag is 0, and not signaled when gfv_matrix_present_flag is 0 or gfv_base_pic_flag is 1. If gfv_matrix_pred_flag does not exist, the value of gfv_matrix_pred_flag can be considered 0.

[0114] If the gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, and gfv_matrix_pred_flag is 1, then there must be a previous generated face video SEI message in the current CLVS that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0115] A generative face video SEI message may include matrix precision information (gfv_matrix_element_precision_factor_minus1). The matrix precision information may represent the precision of a matrix element signaled in the generative face video SEI message. For example, a value obtained by adding 1 to gfv_matrix_element_precision_factor_minus1 may represent the precision of a matrix element signaled in the generative face video SEI message. The value of gfv_matrix_element_precision_factor_minus1 must be in the range from 0 to 31.

[0116] gfv_matrix_element_precision_factor_minus1 may be signaled based on gfv_matrix_pred_flag. For example, gfv_matrix_element_precision_factor_minus1 may be signaled when gfv_matrix_pred_flag is 0, and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_matrix_element_precision_factor_minus1 may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0117] In the case where gfv_matrix_present_flag is 1, gfv_base_pic_flag is 0, and gfv_matrix_pred_flag is 1 in the current generated face video SEI message, the value of gfv_matrix_element_precision_factor_minus1 of the current generated face video SEI message can be considered the same value as the value of gfv_matrix_element_precision_factor_minus1 of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0118] A generative face video SEI message may include matrix type count information (gfv_num_matrix_types_minus1). The matrix type count information may indicate the number of matrix types signaled in the generative face video SEI message. For example, a value obtained by adding 1 to gfv_num_matrix_types_minus1 may indicate the number of matrix types signaled in the generative face video SEI message. The value of gfv_num_matrix_types_minus1 ranges from 0 to 2 6 It must fall within the range up to -1.

[0119] gfv_num_matrix_types_minus1 may be signaled based on gfv_matrix_pred_flag. For example, gfv_num_matrix_types_minus1 may be signaled when gfv_matrix_pred_flag is 0, and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_num_matrix_types_minus1 may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0120] If gfv_matrix_pred_flag is 1 and gfv_base_pic_flag is 0 in the current generated face video SEI message, the value of gfv_num_matrix_types_minus1 must be the same as the value of gfv_num_matrix_types_minus1 of the previous generated face video SEI message in the current CLVS that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0121] In addition, if gfv_matrix_present_flag is 1, gfv_base_pic_flag is 0, and gfv_matrix_pred_flag is 1 in the current generated face video SEI message, the value of gfv_num_matrix_types_minus1 can be considered the same value as gfv_num_matrix_types_minus1 of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0122] A generative face video SEI message may include matrix type information (gfv_matrix_type_idx[i]). The matrix type information may indicate what matrix type the i-th signaled matrix is ​​in the generative face video SEI message. For example, the type of the i-th signaled matrix (hereinafter, the i-th matrix type) can be determined by referring to Table 4 based on the value of gfv_matrix_type_idx[i]. Here, the above i may represent an index for distinguishing each matrix type among a plurality of matrix types included in the generative face video SEI message.

[0123] ValueSpecification0Affine translation matrix with the size of 2*2 or 3*3.1Covariance matrix with size of 2*2 or 3*3.2Mouth matrix representing mouth motion.3Eye matrix representing the open-close status and level of eyes.4Head rotation paramters with the size of 2*2 or 3*3 representing the head rotation in 2D space or 3D space.5Head translation matrix with the size of 1*2 or 1*3 representing head translationin 2D space or 3D space.6Head location matrix with size of 1*2 or 1*3 representing the head location in 2D space or 3D space.7Compact feature matrix with the size being specified by gfv_matrix_width_minus1[i] and gfv_matrix_height_minus1[i].8…31Other matrix that may be used as determined by the application with the size being specified by gfv_matrix_width_minus1[i] and gfv_matrix_height_minus1[i].32…63Reserved

[0124] Referring to Table 4, the value of gfv_matrix_type_idx[i] must be in the range from 0 to 63. However, in the bitstream according to the present disclosure, the value of gfv_matrix_type_idx[i] must be in the range from 0 to 31, and values ​​in the range from 32 to 63 may be reserved for future use. Additionally, the image decoding device according to the present disclosure must allow gfv_matrix_type_idx[i] having a value of 31 or higher to exist in the bitstream, and must ignore all information regarding the i-th matrix type corresponding to gfv_matrix_type_idx[i] having a value of 31 or higher.

[0125] gfv_matrix_type_idx[i] may be signaled based on at least one of gfv_matrix_pred_flag or gfv_num_matrix_types_minus1. For example, gfv_matrix_type_idx[i] may be signaled when gfv_matrix_pred_flag is 0, and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_matrix_type_idx[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag. gfv_matrix_type_idx[i] may be signaled as many times as the number of matrix types signaled in the current generated face video SEI message according to gfv_num_matrix_types_minus1.

[0126] Generative face video SEI messages may include a matrix-keypoint count matching flag (gfv_num_matrices_equal_to_num_kps_flag[i]). The matrix-keypoint count matching flag may indicate whether the number of matrices for the i-th matrix type is equal to the number of keypoints. Here, the keypoint count information (gfv_num_kps_minus1) included in the generative face video SEI message may represent the number of keypoints. For example, the value obtained by adding 1 to gfv_num_kps_minus1 may be the number of keypoints.

[0127] For example, if gfv_num_matrices_equal_to_num_kps_flag[i] is 1, it may indicate that the number of matrices for the i-th signaled matrix type is equal to gfv_num_kps_minus1 plus 1, and if gfv_num_matrices_equal_to_num_kps_flag[i] is 0, it may indicate that the number of matrices for the i-th signaled matrix type is not equal to gfv_num_kps_minus1 plus 1.

[0128] gfv_num_matrices_equal_to_num_kps_flag[i] may be signaled based on at least one of gfv_matrix_pred_flag, gfv_num_matrix_types_minus1, gfv_matrix_type_idx[i], or gfv_coordinate_present_flag. For example, gfv_num_matrices_equal_to_num_kps_flag[i] may be signaled when gfv_matrix_pred_flag is 0 and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_num_matrices_equal_to_num_kps_flag[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag. gfv_num_matrices_equal_to_num_kps_flag[i] can be signaled as many times as the number of matrix types signaled in the current generative face video SEI message according to gfv_num_matrix_types_minus1.

[0129] As another example, gfv_num_matrices_equal_to_num_kps_flag[i] may be signaled when gfv_matrix_type_idx[i] is 0 or 1 and gfv_coordinate_present_flag is 1, and may not be signaled in at least one of the cases where gfv_matrix_type_idx[i] is greater than 1 or gfv_coordinate_present_flag is 0. Here, the keypoint information presence flag (gfv_coordinate_present_flag) included in the generative face video SEI message may indicate whether keypoint coordinate information exists. For example, if gfv_coordinate_present_flag is 1, it may indicate that keypoint coordinate information exists.

[0130] In the case where gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, gfv_matrix_pred_flag is 1, gfv_matrix_type_idx[i] is 0 or 1, and gfv_coordinate_present_flag is 1, the value of gfv_num_matrices_equal_to_num_kps_flag[i] can be considered the same value as gfv_num_matrices_equal_to_num_kps_flag[i] of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0131] Alternatively, if gfv_num_matrices_equal_to_num_kps_flag[i] does not exist, the value of gfv_num_matrices_equal_to_num_kps_flag[i] can be considered 0.

[0132] Generative face video SEI messages may include matrix count derivation information by matrix type (gfv_num_matrices_info[i]). The matrix count derivation information by matrix type may provide information for deriving the number of matrices for the i-th matrix type. The value of gfv_num_matrices_info[i] ranges from 0 to 2 10 It must fall within the range up to -1.

[0133] gfv_num_matrices_info[i] may be signaled based on at least one of gfv_matrix_pred_flag, gfv_num_matrix_types_minus1, or gfv_num_matrices_equal_to_num_kps_flag[i]. For example, gfv_num_matrices_info[i] may be signaled when gfv_matrix_pred_flag is 0 and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_num_matrices_info[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0134] As another example, gfv_num_matrices_info[i] may be signaled when gfv_num_matrices_equal_to_num_kps_flag[i] is 0, and may not be signaled when gfv_num_matrices_equal_to_num_kps_flag[i] is 1. gfv_num_matrices_info[i] may be signaled as many times as the number of matrix types signaled in the current generative face video SEI message according to gfv_num_matrix_types_minus1.

[0135] If gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, gfv_matrix_pred_flag is 1, gfv_matrix_type_idx[i] is 0 or 1, and any of gfv_coordinate_present_flag or gfv_num_matrices_equal_to_num_kps_flag[i] is 0, then the value of gfv_num_matrices_info[i] can be considered the same value as the value of gfv_num_matrices_info[i] in the previous generated face video SEI message in decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0136] The generative face video SEI message may include matrix-related information by matrix type. Here, the matrix-related information by matrix type may include at least one of matrix count information (gfv_num_matrices_minus1[i]) indicating the number of matrices by matrix type, matrix width information (gfv_matrix_width_minus1[i]) indicating the width of the matrix by matrix type, and matrix height information (gfv_matrix_height_minus1[i]) indicating the height of the matrix by matrix type.

[0137] Generative face video SEI messages may include matrix count information (gfv_num_matrices_minus1[i]). This matrix count information may represent the number of matrices for each i-th matrix type. For example, adding 1 to gfv_num_matrices_minus1[i] may represent the number of matrices for each i-th matrix type. The value of gfv_num_matrices_minus1[i] ranges from 0 to 2 10It must fall within the range up to -1.

[0138] gfv_num_matrices_minus1[i] may be signaled based on at least one of gfv_matrix_pred_flag, gfv_num_matrix_types_minus1, or gfv_matrix_type_idx[i]. For example, gfv_num_matrices_minus1[i] may be signaled when gfv_matrix_pred_flag is 0 and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_num_matrices_minus1[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0139] As another example, gfv_num_matrices_minus1[i] may be signaled when gfv_matrix_type_idx[i] is greater than or equal to 7, and may not be signaled when gfv_matrix_type_idx[i] is less than 7. gfv_num_matrices_minus1[i] may be signaled as many times as the number of matrix types signaled in the current generative face video SEI message according to gfv_num_matrix_types_minus1.

[0140] If gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, gfv_matrix_pred_flag is 1, and gfv_matrix_type_idx[i] is greater than or equal to 7, the value of gfv_num_matrices_minus1[i] can be considered the same value as gfv_num_matrices_minus1[i] of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0141] Generative face video SEI messages may include matrix width information (gfv_matrix_width_minus1[i]). The matrix width information may represent the width of the matrix for the i-th matrix type. For example, the value of gfv_matrix_width_minus1[i] plus 1 may represent the width of the matrix for the i-th matrix type. The value of gfv_matrix_width_minus1[i] ranges from 0 to 2 10 It must fall within the range up to -1.

[0142] gfv_matrix_width_minus1[i] may be signaled based on at least one of gfv_matrix_pred_flag, gfv_num_matrix_types_minus1, or gfv_matrix_type_idx[i]. For example, gfv_matrix_width_minus1[i] may be signaled when gfv_matrix_pred_flag is 0 and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_matrix_width_minus1[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0143] As another example, gfv_matrix_width_minus1[i] may be signaled when gfv_matrix_type_idx[i] is equal to 2, equal to 3, or greater than or equal to 7, and may not be signaled when gfv_matrix_type_idx[i] is 0, 1, 4, 5, or 6. gfv_matrix_width_minus1[i] may be signaled as many times as the number of matrix types signaled in the current generative face video SEI message according to gfv_num_matrix_types_minus1.

[0144] If gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, gfv_matrix_pred_flag is 1, and gfv_matrix_type_idx[i] has a value of 2, 3, or is greater than or equal to 7, then the value of gfv_matrix_width_minus1[i] can be considered the same value as the value of gfv_matrix_width_minus1[i] of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0145] Generative face video SEI messages may include matrix height information (gfv_matrix_height_minus1[i]). The matrix height information may represent the height of the matrix for the i-th matrix type. For example, the value of gfv_matrix_height_minus1[i] plus 1 may represent the height of the matrix for the i-th matrix type. The value of gfv_matrix_height_minus1[i] ranges from 0 to 2 10 It must fall within the range up to -1.

[0146] gfv_matrix_height_minus1[i] may be signaled based on at least one of gfv_matrix_pred_flag, gfv_num_matrix_types_minus1, or gfv_matrix_type_idx[i]. For example, gfv_matrix_height_minus1[i] may be signaled when gfv_matrix_pred_flag is 0, and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_matrix_height_minus1[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0147] As another example, gfv_matrix_height_minus1[i] may be signaled when gfv_matrix_type_idx[i] is equal to 2, equal to 3, or greater than or equal to 7, and may not be signaled when gfv_matrix_type_idx[i] is 0, 1, 4, 5, or 6. gfv_matrix_height_minus1[i] may be signaled as many times as the number of matrix types signaled in the current generative face video SEI message according to gfv_num_matrix_types_minus1.

[0148] If gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, gfv_matrix_pred_flag is 1, and gfv_matrix_type_idx[i] has a value of 2, 3, or is greater than or equal to 7, then the value of gfv_matrix_height_minus1[i] can be considered the same value as the value of gfv_matrix_height_minus1[i] of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0149] Generative face video SEI messages may include a 3D matrix flag (gfv_matrix_for_3D_space_flag[i]). The 3D matrix flag may indicate whether a matrix of type i is a matrix defined in 3D space. For example, if gfv_matrix_for_3D_space_flag[i] is 1, it may indicate that a matrix of type i is a matrix defined in 3D space, and if gfv_matrix_for_3D_space_flag[i] is 0, it may indicate that a matrix of type i is a matrix defined in 2D space.

[0150] gfv_matrix_for_3D_space_flag[i] may be signaled based on at least one of gfv_matrix_pred_flag, gfv_num_matrix_types_minus1, gfv_matrix_type_idx[i], or gfv_coordinate_present_flag. For example, gfv_matrix_for_3D_space_flag[i] may be signaled when gfv_matrix_pred_flag is 0 and may not be signaled when gfv_matrix_pred_flag is 1. In this case, gfv_matrix_for_3D_space_flag[i] may not be signaled when gfv_matrix_pred_flag is 1, regardless of the value of gfv_base_pic_flag.

[0151] As another example, gfv_matrix_for_3D_space_flag[i] may be signaled when gfv_matrix_type_idx[i] is 4, 5, or 6 and gfv_coordinate_present_flag is 0, and may not be signaled when gfv_matrix_type_idx[i] is less than 4, gfv_matrix_type_idx[i] is greater than 6, or gfv_coordinate_present_flag is 1. gfv_matrix_for_3D_space_flag[i] may be signaled as many times as the number of matrix types signaled in the current generative face video SEI message according to gfv_num_matrix_types_minus1.

[0152] In the case where gfv_matrix_present_flag included in the current generated face video SEI message is 1, gfv_base_pic_flag is 0, gfv_matrix_pred_flag is 1, gfv_matrix_type_idx[i] is 4, 5, or 6, and gfv_coordinate_present_flag is 0, the value of gfv_matrix_for_3D_space_flag[i] can be considered the same value as the value of gfv_matrix_for_3D_space_flag[i] of the previous generated face video SEI message in the decoding order that has the same gfv_id as the current generated face video SEI message and has a gfv_base_pic_flag of 1.

[0153] Generated face video SEI messages may be included in a network abstraction layer (NAL) unit of a bitstream, but are not limited thereto. For example, the generated face video information according to the present disclosure may be configured in a high-level syntax of the bitstream. Here, the high-level syntax may be at least one of a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), or a slice header (SH). Alternatively, the generated face video SEI message according to the present disclosure may be defined as a separate NAL unit type within the bitstream.

[0154] FIG. 5 illustrates a schematic configuration of a decoding device (300) that performs a method for restoring a video picture according to the present disclosure.

[0155] Referring to FIG. 5, the decoding device (300) may include a receiving unit (500), a video information extraction unit (510), and a video restoration unit (520).

[0156] The receiver (500) can receive a bitstream including an encoded video picture.

[0157] The video information extraction unit (510) can extract video information regarding an encoded video picture from the bitstream. Additionally, the video information extraction unit (710) can extract generated face video information from the bitstream, as seen with reference to FIG. 4.

[0158] The video restoration unit (520) can restore an encoded video picture based on extracted video information.

[0159] FIG. 6 illustrates a method for generating a bitstream performed in an encoding device (200) according to the present disclosure.

[0160] A video picture being encoded can be received (S600).

[0161] Video information regarding the video picture can be generated by encoding the received video picture (S610).

[0162] A bitstream containing video information about a video picture can be generated (S620).

[0163] In addition, generative face video information applied to the bitstream can be generated, as seen with reference to FIG. 4. The generative face video information can be included in the bitstream.

[0164] FIG. 7 illustrates a schematic configuration of an encoding device (200) that performs a method for generating a bitstream according to the present disclosure.

[0165] Referring to FIG. 7, the encoding device (200) may include a receiving unit (700), a video compression unit (710), and a bitstream generation unit (720).

[0166] The receiver (700) can receive one or more video pictures that are encoded.

[0167] The video compression unit (710) can generate video information regarding the video picture by encoding one or more received video pictures. The video compression unit (710) can generate generative face video information applied to the bitstream.

[0168] The bitstream generation unit (720) can generate a bitstream including the video information. The bitstream generation unit (720) can generate a bitstream that further includes the generated generated face video information.

[0169] In the embodiments described above, methods are described based on flowcharts as a series of steps or blocks; however, the embodiments are not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps as described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, and other steps may be included, or one or more steps of the flowcharts may be omitted without affecting the scope of the embodiments of this document.

[0170] The method according to the embodiments of the present document described above may be implemented in the form of software, and the encoding device and / or decoding device according to the present document may be included in a device that performs image processing, such as a TV, computer, smartphone, set-top box, display device, etc.

[0171] When the embodiments described in this document are implemented in software, the method described above may be implemented as a module (process, function, etc.) that performs the function described above. The module may be stored in memory and executed by a processor. The memory may be located inside or outside the processor and may be connected to the processor by various well-known means. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices. That is, the embodiments described in this document may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information on instructions or algorithms for implementation may be stored on a digital storage medium.

[0172] In addition, the decoding device and encoding device to which the embodiment(s) of the present specification are applied may be included in multimedia broadcasting transmission and reception devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video conversation devices, real-time communication devices such as video communication, mobile streaming devices, storage media, camcorders, Video on Demand (VoD) service providers, Over-the-top video (OTT) devices, internet streaming service providers, 3D video devices, virtual reality (VR) devices, augmented reality (AR) devices, video phone video devices, transportation terminals (e.g., vehicle terminals (including autonomous vehicles), airplane terminals, ship terminals, etc.), and medical video devices, and may be used to process video signals or data signals. For example, Over-the-top video (OTT) devices may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, Digital Video Recorders (DVRs), etc.

[0173] Additionally, the processing method to which the embodiment(s) of this specification are applied may be produced in the form of a program that is executed by a computer and may be stored on a computer-readable recording medium. Multimedia data having a data structure according to the embodiment(s) of this specification may also be stored on a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Blu-ray disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. Additionally, the computer-readable recording medium includes a medium implemented in the form of a carrier wave (e.g., transmission over the Internet). Furthermore, a bitstream generated by an encoding method may be stored on a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0174] Additionally, the embodiments of this specification may be implemented as a computer program product by program code, and said program code may be executed on a computer by the embodiments of this specification. said program code may be stored on a carrier readable by a computer.

[0175] FIG. 8 shows an example of a content streaming system to which embodiments of the present disclosure can be applied.

[0176] Referring to FIG. 8, a content streaming system to which the embodiment(s) of the present specification are applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0177] The above encoding server compresses content input from multimedia input devices, such as smartphones, cameras, and camcorders, into digital data to generate a bitstream and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and camcorders, generate the bitstream directly, the encoding server may be omitted.

[0178] The bitstream above may be generated by an encoding method or a bitstream generation method to which the embodiment(s) of the present specification are applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0179] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server plays the role of controlling commands and responses between each device within the content streaming system.

[0180] The streaming server may receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content may be received in real time. In this case, to provide a seamless streaming service, the streaming server may store the bitstream for a certain period of time.

[0181] Examples of the above user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0182] Each server within the above-mentioned content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0183] The claims described in this specification may be combined in various ways. For example, the technical features of the method claims in this specification may be combined to be implemented as a device, and the technical features of the device claims in this specification may be combined to be implemented as a method. Furthermore, the technical features of the method claims and the technical features of the device claims in this specification may be combined to be implemented as a device, and the technical features of the method claims and the technical features of the device claims in this specification may be combined to be implemented as a method.

Claims

1. A step of receiving a bitstream including an encoded video picture; and The method includes the step of restoring an encoded video picture contained in the bitstream, The above bitstream includes a generated face video SEI (Supplementary Enhancement Information) message, and The above-mentioned generative face video SEI message includes at least one of a reference picture flag indicating whether the currently decoded output picture corresponds to a reference picture, or a matrix prediction flag indicating whether matrix element difference value information exists. A method in which the above-mentioned generative face video SEI message is obtained from a NAL (network abstraction layer) unit of the bitstream.

2. In Paragraph 1, If the value of the above matrix prediction flag is 0, it indicates that matrix element value information exists, and If the value of the above matrix prediction flag is 1, it indicates that matrix element difference value information exists, and The above matrix element value information includes at least one of matrix element integer value information or matrix element decimal value information, and A method wherein the matrix element difference value information comprises at least one of matrix element integer difference value information or matrix element fractional difference value information.

3. In Paragraph 1, The above-mentioned generated face video SEI message includes matrix precision information indicating the precision of matrix elements, and A method in which the above matrix precision information is signaled based on the above matrix prediction flag.

4. In Paragraph 3, If the value of the above matrix prediction flag is 0, the above matrix precision information is signaled, and A method in which, when the value of the matrix prediction flag is 1, the matrix precision information is not signaled regardless of the value of the reference picture flag.

5. In Paragraph 1, The above-mentioned generative face video SEI message includes matrix type count information indicating the number of matrix types, and The method of signaling the number of matrix types above based on the matrix prediction flag.

6. In Paragraph 5, If the value of the above matrix prediction flag is 0, the above matrix type count information is signaled, and A method in which, when the value of the matrix prediction flag is 1, the matrix type count information is not signaled regardless of the value of the reference picture flag.

7. In Paragraph 5, The above-mentioned generated face video SEI message includes matrix type information indicating the type of matrix included in the above-mentioned generated face video SEI message, and A method in which the above matrix type information is signaled based on at least one of the above matrix prediction flag or the above matrix type count information.

8. In Paragraph 7, If the value of the above matrix prediction flag is 0, the above matrix type information is signaled, and A method in which, when the value of the matrix prediction flag is 1, the matrix type information is not signaled regardless of the value of the reference picture flag.

9. In Paragraph 7, The above-mentioned generative face video SEI message includes matrix-related information by matrix type, and The matrix-related information for each matrix type is signaled based on at least one of the matrix prediction flag or the matrix type information, and A method comprising at least one of the above matrix-related information, matrix count information indicating the number of matrices by matrix type, matrix width information indicating the width of the matrix by matrix type, or matrix height information indicating the height of the matrix by matrix type.

10. In Paragraph 9, If the value of the above matrix prediction flag is 0, matrix-related information for each matrix type is signaled, and A method in which, when the value of the above matrix prediction flag is 1, matrix-related information for each matrix type is not signaled regardless of the value of the above reference picture flag.

11. In Paragraph 9, If the value of the above matrix type information is greater than or equal to 7, the above matrix count information, the above matrix width information, and the above matrix height information are signaled, and A method in which, when the value of the matrix type information is 2 or 3, the matrix width information and the matrix height information are signaled, and the matrix count information is not signaled.

12. A step of receiving a video picture to be encoded; A step of encoding the received video picture to generate video information regarding the video picture; A step of generating a generative face video SEI (Supplementary Enhancement Information) message; and The method includes the step of generating a bitstream including the video information and the generated face video SEI message, The above-mentioned generative face video SEI message includes at least one of a reference picture flag indicating whether the currently decoded output picture corresponds to a reference picture, or a matrix prediction flag indicating whether matrix element difference value information exists. A method in which the above-mentioned generative face video SEI message is encoded into the NAL (network abstraction layer) unit of the bitstream.

13. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 12.

14. A step of generating a bitstream; wherein the bitstream is generated based on the steps of receiving a video picture to be encoded, encoding the received video picture to generate video information regarding the video picture, and generating a generative face video SEI (Supplementary Enhancement Information) message, and The method includes the step of transmitting data including the above bitstream, The above-mentioned generative face video SEI message includes at least one of a reference picture flag indicating whether the currently decoded output picture corresponds to a reference picture, or a matrix prediction flag indicating whether matrix element difference value information exists. A method in which the above-mentioned generative face video SEI message is encoded into the NAL (network abstraction layer) unit of the bitstream.

Images