Video encoding / decoding method, and method and device for transmitting bitstream

The proposed method improves encoding/decoding efficiency and accuracy by integrating neural network-based intra-prediction with additional modes to manage high-resolution video data, addressing the increased costs and volume challenges.

WO2026235424A1PCT designated stage Publication Date: 2026-11-12HYUNDAI MOTOR CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2026/096036
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-07
Filing Date
2026-04-17
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

The increasing demand for high-resolution, high-quality video has led to a surge in video data volume, resulting in higher transmission and storage costs, necessitating improved encoding and decoding technologies, particularly for neural network-based intra prediction methods.

Method used

An image encoding/decoding method that combines neural network-based intra-prediction with additional prediction modes, utilizing a neural network model to generate prediction blocks by inputting context from adjacent samples and applying weighted sums of multiple prediction blocks.

Benefits of technology

Enhances encoding/decoding efficiency and accuracy by leveraging neural networks for intra-prediction, reducing data volume and costs associated with high-resolution video transmission and storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2026096036_12112026_PF_FP_ABST
    Figure KR2026096036_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Provided are a video encoding / decoding method, a device, a recording medium storing a bitstream, and a transmission method. The video decoding method comprises the steps of: deriving a prediction mode of a current block; generating a first prediction block of the current block on the basis of the prediction mode of the current block; deriving a context of the current block on the basis of reference samples adjacent to the current block; generating a second prediction block of the current block by inputting the context of the current block into a neural network model; and generating a final prediction block of the current block on the basis of a weighted sum of the first prediction block and the second prediction block.
Need to check novelty before this filing date? Find Prior Art

Description

Video encoding / decoding method, method for transmitting bitstream and device

[0001] The present disclosure relates to an image encoding / decoding method, a method for transmitting a bitstream, and an apparatus. Specifically, the present disclosure relates to an image encoding / decoding method that performs prediction by combining a neural network-based intra-prediction mode and an additional prediction mode, a method for transmitting a bitstream, and an apparatus.

[0002] Recently, the demand for high-resolution, high-quality video, such as UHD (Ultra High Definition) video, has been increasing across various application fields. As video data becomes higher in resolution and quality, the relative volume of data increases compared to conventional video data; consequently, transmission and storage costs increase when video data is transmitted using existing wired or wireless broadband lines or stored using existing storage media. To address these issues arising from the increase in video data resolution and quality, high-efficiency video encoding and decoding technologies for video with higher resolution and quality are required.

[0003] Neural Network-based Intra Prediction (NNIP), used in video encoding / decoding technology, is a method that inputs reconstructed samples surrounding the current block into a neural network model and generates a predicted block based on the output value of the neural network model. In other words, neural network-based intra prediction can refer to an intra prediction technology that utilizes a neural network model. A method is required to improve the accuracy and encoding efficiency of the neural network-based intra prediction method.

[0004] The present disclosure aims to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] In addition, the present disclosure aims to provide a recording medium storing a bitstream generated by an image decoding method or device according to the present disclosure.

[0006] In addition, the present disclosure aims to provide an image encoding / decoding method and apparatus that performs prediction of the current block by combining a neural network-based intra-prediction value and a prediction value based on a separate prediction mode.

[0007] A video decoding method according to one embodiment of the present disclosure may include the steps of: deriving a prediction mode of a current block; generating a first prediction block of the current block based on the prediction mode of the current block; deriving a context of the current block based on a reference sample adjacent to the current block; inputting the context of the current block into a neural network model to generate a second prediction block of the current block; and generating a final prediction block of the current block based on a weighted sum of the first prediction block and the second prediction block.

[0008] In the above image decoding method, the prediction mode of the current block may be an intra prediction mode derived by the DIMD (Decoder side Intra Mode Derivation) method.

[0009] In the above image decoding method, the first prediction block can be generated by combining prediction blocks generated based on each of a plurality of intra prediction modes derived by the DIMD method.

[0010] In the above image decoding method, the prediction mode of the current block may be an intra prediction mode derived by the TIMD (Template-based Intra Mode Derivation) method.

[0011] In the above image decoding method, the first prediction block can be generated by combining prediction blocks generated based on each of a plurality of intra prediction modes derived by the TIMD method.

[0012] In the above image decoding method, the first prediction block may be a prediction block generated based on a block indicated by the block vector of the current block.

[0013] In the above image decoding method, the first prediction block can be generated based on a combination of a reference sample line and an intra prediction mode selected from a combination of a plurality of reference sample lines and intra prediction modes.

[0014] In the above image decoding method, the first prediction block can be generated by performing inter-prediction on the current block.

[0015] In the above image decoding method, the first prediction block may be a prediction block for a first partition divided from the current block, and the second prediction block may be a prediction block for a second partition divided from the current block.

[0016] In the above image decoding method, the first prediction block can be generated based on information derived by inputting the context of a surrounding block adjacent to the current block into a neural network model.

[0017] In the above image decoding method, the first prediction block can be generated by inputting a context derived based on reference samples of a region adjacent to the current block into the neural network model.

[0018] In the above image decoding method, the final prediction block of the current block is generated by applying a first weight and a second weight to the first prediction block and the second prediction block, respectively, and the first weight and the second weight can be derived based on the distortion value between the predicted value and the restored value of the template area of ​​the first prediction block and the distortion value between the predicted value and the restored value of the template area of ​​the second prediction block.

[0019] In the above image decoding method, the region adjacent to the current block is determined to be one of a first candidate reference region including samples adjacent to the left of the current block, a second candidate reference region including samples adjacent to the top of the current block, and a third candidate reference region including samples adjacent to the left of the current block and samples adjacent to the top of the current block, and the template region of the first prediction block may be characterized as corresponding to the region adjacent to the current block.

[0020] A video encoding method according to one embodiment of the present disclosure comprises: a step of deriving a prediction mode of a current block; a step of generating a first prediction block of the current block based on the prediction mode of the current block; a step of deriving a context of the current block based on a reference sample adjacent to the current block; a step of inputting the context of the current block into a neural network model to generate a second prediction block of the current block; and

[0021] It may include the step of generating a final prediction block of the current block based on the weighted sum of the first prediction block and the second prediction block.

[0022] A method for transmitting a bitstream generated by an image encoding method according to one embodiment of the present disclosure may transmit a bitstream generated by the image encoding method, comprising the steps of: transmitting a bitstream; deriving a prediction mode of a current block; generating a first prediction block of the current block based on the prediction mode of the current block; deriving a context of the current block based on a reference sample adjacent to the current block; inputting the context of the current block into a neural network model to generate a second prediction block of the current block; and generating a final prediction block of the current block based on a weighted sum of the first prediction block and the second prediction block.

[0023] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0024] According to the present disclosure, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency may be provided.

[0025] In addition, the present disclosure may provide an image encoding / decoding method and apparatus that improves the accuracy of prediction by utilizing neural network-based intra-predicted values.

[0026] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0027] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the technologies of the present disclosure.

[0028] Figure 2 is a diagram illustrating a method for dividing blocks using a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0029] FIGS. 3a and 3b are diagrams showing a plurality of intra prediction modes including wide-angle intra prediction modes.

[0030] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0031] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the technologies of the present disclosure.

[0032] FIG. 6 is a diagram illustrating a neural network-based intra-prediction method according to one embodiment of the present disclosure.

[0033] FIG. 7 is a drawing for explaining a template-based intra-prediction mode induction method according to one embodiment of the present disclosure.

[0034] FIG. 8 is a drawing for explaining an intra-template matching mode prediction method according to one embodiment of the present disclosure.

[0035] FIG. 9 is a diagram illustrating a template-based multi-reference sample line intra prediction method according to one embodiment of the present disclosure.

[0036] FIG. 10 is a diagram illustrating a combination between inter-prediction and intra-prediction that may occur in a geometric division mode according to one embodiment of the present disclosure.

[0037] FIG. 11 is a diagram illustrating an intra-block copy-based prediction method according to one embodiment of the present disclosure.

[0038] FIGS. 12A to 12C are drawings for illustrating candidates for the shape of a reference region according to one embodiment of the present disclosure.

[0039] FIG. 13 is a flowchart illustrating an image decoding method according to one embodiment of the present disclosure.

[0040] Some embodiments of the present disclosure will be described in detail below with reference to the exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the embodiments, if it is determined that a detailed description of related known components or functions could obscure the essence of the embodiments, such detailed description is omitted.

[0041] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the technologies of the present disclosure. Hereinafter, the image encoding device and its sub-components will be described with reference to FIG. 1.

[0042] The video encoding device may be configured to include a picture splitting unit (110), a prediction unit (120), a subtractor (130), a conversion unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse conversion unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0043] Each component of the video encoding device may be implemented in hardware or software, or as a combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the software function corresponding to each component.

[0044] A single image (video) consists of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). And each CTU is divided into one or more Coding Units (CUs) by a tree structure. Information applicable to each CU is encoded as the syntax of the CU, and information applicable to all CUs included in a single CTU is encoded as the syntax of the CTU. Additionally, information applicable to all blocks within a single slice is encoded as the syntax of the slice header, and information applicable to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). Also, information commonly referenced by one or more SPSs is encoded in a Video Parameter Set (VPS). Additionally, information commonly applicable to a single tile or tile group may be encoded as the syntax of a tile or tile group header. The syntax included in the SPS, PPS, slice header, and tile or tile group header may be referred to as high-level syntax.

[0045] The picture splitting unit (110) determines the size of the CTU. Information regarding the size of the CTU (CTU size) is encoded as a syntax of SPS or PPS and transmitted to an image decoding device.

[0046] The picture division unit (110) divides each picture constituting the image into multiple CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. The leaf nodes in the tree structure become the CUs, which are the basic units of encoding.

[0047] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of equal size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that combines two or more of these QT, BT, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT combined may be referred to as an MTT (Multiple-Type Tree).

[0048] Figure 2 is a diagram illustrating a method for dividing blocks using a QTBTTT structure.

[0049] As illustrated in FIG. 2, the CTU can first be split into a QT structure. Quadtree splitting can be repeated until the size of the splitting block reaches the minimum block size of the leaf node allowed in QT (MinQTSize). A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoder (155) and signaled to the image decoder. If the leaf node of the QT is not larger than the maximum block size of the root node allowed in BT (MaxBTSize), it can be further split into one or more of the BT structure or TT structure. In the BT structure and / or TT structure, multiple splitting directions may exist. For example, there may be two directions in which the block of the corresponding node is split horizontally and vertically. As shown in Figure 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether the nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or the splitting type (binary or ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0050] Alternatively, prior to encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of the lower layer, the CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the value of the CU split flag (split_cu_flag) indicates that it is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a coding unit (CU), which is the basic unit of encoding. If the value of the CU split flag (split_cu_flag) indicates that it is split, the video encoding device starts encoding from the first flag in the manner described above.

[0051] When QTBT is used as another example of a tree structure, there may be two types: a type that divides the block of the corresponding node horizontally into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that divides it vertically (i.e., symmetric vertical splitting). A splitting flag (split_flag) indicating whether each node of the BT structure is split into a block of a lower layer and splitting type information indicating the type of splitting are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may also be an additional type that divides the block of the corresponding node into two blocks of an asymmetric shape. The asymmetric shape may include a shape that divides the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that divides the block of the corresponding node diagonally.

[0052] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from a CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of QTBTTT) is referred to as the 'current block'. Depending on the adoption of QTBTTT partitioning, the shape of the current block may be not only square but also rectangular.

[0053] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0054] Generally, current blocks within a picture can each be predictively coded. Typically, the prediction of a current block can be performed using an intra-prediction technique (using data from the picture containing the current block) or an inter-prediction technique (using data from a picture coded prior to the picture containing the current block). Inter-prediction includes both unidirectional and bidirectional prediction.

[0055] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture containing the current block. Multiple intra prediction modes exist depending on the prediction direction. For example, as shown in FIG. 3a, multiple intra prediction modes may include two non-directional modes, including Planar mode and DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently for each prediction mode.

[0056] For efficient directional prediction for a rectangular current block, directional modes (intra-prediction modes 67 through 80 and -1 through -14) illustrated by dashed arrows in FIG. 3b may be additionally used. These may be referred to as "wide angle intra-prediction modes." In FIG. 3b, the arrows indicate corresponding reference samples used for prediction and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. Among the wide angle intra-prediction modes, some wide angle intra-prediction modes available for the current block may be determined by the ratio of the width to the height of the rectangular current block. For example, wide-angle intra-prediction modes with an angle less than 45 degrees (intra-prediction modes 67 to 80) are available when the current block is a rectangular shape with a height less than the width, and wide-angle intra-prediction modes with an angle greater than -135 degrees (intra-prediction modes -1 to -14) are available when the current block is a rectangular shape with a width greater than the height.

[0057] The intra prediction unit (122) can determine the intra prediction mode to use for encoding the current block. In some examples, the intra prediction unit (122) may encode the current block using several intra prediction modes and select an appropriate intra prediction mode to use from the tested modes. For example, the intra prediction unit (122) may calculate the rate-distortion values ​​using a rate-distortion analysis of several tested intra prediction modes and select the intra prediction mode having the best rate-distortion features among the tested modes.

[0058] The intra prediction unit (122) selects one intra prediction mode among a plurality of intra prediction modes and predicts the current block using a calculation formula and surrounding pixels (reference pixels) determined according to the selected intra prediction mode. Information regarding the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0059] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for the block most similar to the current block within a reference picture that is encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, it generates a motion vector (MV) corresponding to the displacement between the current block in the current picture and the prediction block in the reference picture. Generally, motion estimation is performed on the lumina component, and the motion vector calculated based on the lumina component is used for both the lumina component and the chroma component. Motion information including information about the reference picture used to predict the current block and information about the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0060] The inter prediction unit (124) may perform interpolation on a reference picture or reference block to increase the accuracy of the prediction. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including those two integer samples. When the process of searching for the block most similar to the current block is performed for the interpolated reference picture, the motion vector can be expressed with precision in fractional units rather than precision in integer sample units. The precision or resolution of the motion vector can be set differently for each unit of the target area to be encoded, such as slice, tile, CTU, CU, etc. When such Adaptive Motion Vector Resolution (AMVR) is applied, information regarding the motion vector resolution to be applied to each target area must be signaled for each target area. For example, if the target area is a CU, information regarding the motion vector resolution applied to each CU is signaled. The information regarding the motion vector resolution may be information indicating the precision of the difference motion vector described later.

[0061] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors representing the block location most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from the reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and generates a first reference block and a second reference block by searching for a block similar to the current block within each reference picture. Then, it generates a prediction block for the current block by averaging or weighting the first reference block and the second reference block. Then, it transmits motion information containing information about the two reference pictures used to predict the current block and information about the two motion vectors to the entropy encoding unit (155). Here, reference picture list 0 consists of restored pictures that are prior to the current picture in the display order, and reference picture list 1 may consist of restored pictures that are prior to the current picture in the display order. However, this is not necessarily limited to this, and restored pictures prior to the current picture in the display order may be additionally included in reference picture list 0, and conversely, restored pictures prior to the current picture may be additionally included in reference picture list 1.

[0062] Various methods can be used to minimize the amount of bits required to encode motion information.

[0063] For example, if the reference picture and motion vector of the current block are identical to the reference picture and motion vector of a neighboring block, the motion information of the current block can be transmitted to an image decoder by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0064] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0065] As for the surrounding blocks for deriving merge candidates, as shown in FIG. 4, all or part of the left block (A0), bottom-left block (A1), top block (B0), top-right block (B1), and top-left block (B2) adjacent to the current block within the current picture may be used. Additionally, a block located within a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the current block is located may be used as a merge candidate. For example, a block located at the same position as the current block within the reference picture (co-located block) or a block adjacent to that same position may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0066] The inter prediction unit (124) constructs a merge list containing a predetermined number of merge candidates using these surrounding blocks. Among the merge candidates included in the merge list, it selects a merge candidate to be used as movement information for the current block and generates merge index information to identify the selected candidate. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0067] Merge skip mode is a special case of merge mode; after quantization, when all transform coefficients for entropy coding are close to zero, only neighbor block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high coding efficiency can be achieved in images with minimal motion, still images, and screen content images.

[0068] Hereinafter, merge mode and merge skip mode will be collectively referred to as merge / skip mode.

[0069] Another method for encoding motion information is the AMVP (Advanced Motion Vector Prediction) mode.

[0070] In AMVP mode, the inter-prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using the surrounding blocks of the current block. As surrounding blocks used to derive predicted motion vector candidates, all or part of the left block (A0), bottom-left block (A1), top block (B0), top-right block (B1), and top-left block (B2) adjacent to the current block within the current picture shown in FIG. 4 may be used. Additionally, blocks located within a reference picture (which may be the same as or different from the reference picture used to predict the current block) other than the current picture where the current block is located may be used as surrounding blocks to derive predicted motion vector candidates. For example, blocks located at the same position as the current block within the reference picture (co-located blocks) or blocks adjacent to the blocks at the same position may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0071] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of the surrounding blocks and determines a predicted motion vector for the current block's motion vector using the predicted motion vector candidates. Then, it calculates a difference motion vector by subtracting the predicted motion vector from the current block's motion vector.

[0072] Predicted motion vectors can be obtained by applying a predefined function (e.g., median, mean operation, etc.) to the predicted motion vector candidates. In this case, the image decoder is also aware of the predefined function. Furthermore, since the surrounding blocks used to derive the predicted motion vector candidates have already been encoded and decoded, the image decoder is also aware of the motion vectors of those surrounding blocks. Therefore, the image decoder does not need to encode information to identify the predicted motion vector candidates. Consequently, in this case, information regarding the difference motion vector and the reference picture used to predict the current block is encoded.

[0073] Meanwhile, the predicted motion vector may be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the difference motion vector and information about the reference picture used to predict the current block.

[0074] The subtractor (130) generates a residual block by subtracting the prediction block generated by the intra prediction unit (122) or the inter prediction unit (124) from the current block.

[0075] The conversion unit (140) converts residual signals within a residual block having pixel values ​​in a spatial domain into conversion coefficients in the frequency domain. The conversion unit (140) can convert the residual signals within the residual block using the entire size of the residual block as the conversion unit, or it can divide the residual block into multiple sub-blocks and use the sub-blocks as the conversion unit to perform the conversion. Alternatively, it can divide the residual signals into two sub-blocks, a conversion area and a non-conversion area, and use only the conversion area sub-block as the conversion unit to convert the residual signals. Here, the conversion area sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only the sub-block has been converted, direction (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. Additionally, the size of the converted area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0076] Meanwhile, the transformation unit (140) can perform transformations on the residual block individually in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices may be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one pair of transformation functions with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) regarding the selected pair of transformation functions among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoder.

[0077] The quantization unit (145) quantizes the transformation coefficients output from the transformation unit (140) using quantization parameters and outputs the quantized transformation coefficients to the entropy encoding unit (155). The quantization unit (145) may quantize the associated residual block directly without transformation for any block or frame. The quantization unit (145) may apply different quantization coefficients (scaling values) depending on the position of the transformation coefficients within the transformation block. The quantization matrix applied to the quantized transformation coefficients arranged in two dimensions can be encoded and signaled to an image decoder.

[0078] The reordering unit (150) can perform reordering of coefficient values ​​for quantized residual values.

[0079] The reordering unit (150) can convert a two-dimensional coefficient array into a one-dimensional coefficient sequence using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from DC coefficients to coefficients in the high-frequency range using a zig-zag scan or a diagonal scan. Depending on the size of the conversion unit and the intra-prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction and a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of a zig-zag scan. That is, depending on the size of the conversion unit and the intra-prediction mode, the scanning method to be used among a zig-zag scan, a diagonal scan, a vertical scan, and a horizontal scan may be determined.

[0080] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transformation coefficients output from the reordering unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0081] Additionally, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the video decoder can divide the block in the same way as the video encoding unit. Additionally, the entropy encoding unit (155) encodes information regarding a prediction type indicating whether the current block is encoded by intra prediction or by inter prediction, and encodes intra prediction information (i.e., information regarding the intra prediction mode) or inter prediction information (information regarding the encoding mode of motion information (merge mode or AMVP mode), the merge index in the case of merge mode, and the reference picture index and difference motion vector in the case of AMVP mode) according to the prediction type. Additionally, the entropy encoding unit (155) encodes information related to quantization, i.e., information regarding quantization parameters and information regarding the quantization matrix.

[0082] The inverse quantization unit (160) inversely quantizes the quantized transformation coefficients output from the quantization unit (145) to generate transformation coefficients. The inverse transformation unit (165) converts the transformation coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0083] The adder (170) restores the current block by adding the restored residual block and the prediction block generated by the prediction unit (120). The pixels within the restored current block are used as reference pixels when intra-predicting the next block in sequence.

[0084] The loop filter section (180) performs filtering on the restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. caused by block-based prediction and transformation / quantization. The loop filter section (180) may include all or part of a deblocking filter (182), a SAO (Sample Adaptive Offset) filter (184), and an ALF (Adaptive Loop Filter, 186) as an in-loop filter.

[0085] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-unit encoding / decoding, and the SAO filter (184) and ALF (186) perform additional filtering on the deblocking filtered image. The SAO filter (184) and ALF (186) are filters used to compensate for the difference between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in CTU units. In contrast, the ALF (186) performs block-unit filtering, and compensates for distortion by applying different filters by distinguishing the degree of edge and change of the corresponding block. Information regarding the filter coefficients to be used in the ALF can be encoded and signaled to an image decoder.

[0086] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in memory (190). Once all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0087] The video encoding device can store the bitstream of encoded video data on a non-transient recording medium or transmit it to a video decoding device using a communication network.

[0088] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the technologies of the present disclosure. Hereinafter, the image decoding device and its sub-components will be described with reference to FIG. 5.

[0089] The image decoding device may be configured to include an entropy decoding unit (510), a reordering unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0090] Similar to the image encoding device of FIG. 1, each component of the image decoding device may be implemented in hardware or software, or in combination of hardware and software. Additionally, the function of each component may be implemented in software, and a microprocessor may be implemented to execute the function of the software corresponding to each component.

[0091] The entropy decoding unit (510) determines the current block to be decoded by decoding the bitstream generated by the video encoding device and extracting information related to block division, and extracts prediction information, information on residual signals, etc., necessary to restore the current block.

[0092] The entropy decoding unit (510) extracts information about the CTU size from the SPS (Sequence Parameter Set) or PPS (Picture Parameter Set) to determine the size of the CTU and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and divides the CTU using the tree structure by extracting division information for the CTU.

[0093] For example, when splitting a CTU using a QTBTTT structure, first, a first flag (QT_split_flag) related to QT splitting is extracted to split each node into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, a second flag (mtt_split_flag) related to MTT splitting and splitting direction (vertical / horizontal) and / or splitting type (binary / ternary) information are extracted to split the corresponding leaf nodes into an MTT structure. Accordingly, each node below the leaf nodes of QT is recursively split into a BT or TT structure.

[0094] As another example, when splitting a CTU using the QTBTTT structure, a CU splitting flag (split_cu_flag) indicating whether to split the CU is first extracted, and if the block is split, a first flag (QT_split_flag) is extracted. During the splitting process, each node may undergo zero or more iterative MTT splitting after zero or more iterative QT splittings. For example, the CTU may undergo MTT splitting immediately, or conversely, only multiple QT splittings may occur.

[0095] As another example, when splitting a CTU using a QTBT structure, a first flag (QT_split_flag) related to the splitting of QT is extracted to split each node into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and split direction information are extracted.

[0096] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded using the division of the tree structure, it extracts information regarding the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-predicted, the entropy decoding unit (510) extracts syntax elements for the intra-predicted information (intra-predicted mode) of the current block. If the prediction type information indicates inter-predicted, the entropy decoding unit (510) extracts syntax elements for the inter-predicted information, namely information indicating the motion vector and the reference picture that the motion vector refers to.

[0097] Additionally, the entropy decoding unit (510) extracts information regarding quantization-related information and information regarding residual signals, as well as information regarding the quantized transformation coefficients of the current block.

[0098] The reordering unit (515) can change the sequence of one-dimensional quantized transformation coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning order performed by the image encoding device.

[0099] The inverse quantization unit (520) inversely quantizes the quantized transformation coefficients and inversely quantizes the quantized transformation coefficients using quantization parameters. The inverse quantization unit (520) may apply different quantization coefficients (scaling values) to the quantized transformation coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transformation coefficients.

[0100] The inverse transformation unit (530) generates a residual block for the current block by inversely transforming the inversely quantized transformation coefficients from the frequency domain to the spatial domain and restoring the residual signals.

[0101] Additionally, when the inverse transformation unit (530) inversely transforms only a part of the transformation block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only the sub-block of the transformation block has been transformed, information on the directionality (vertical / horizontal) of the sub-block (cu_sbt_horizontal_flag) and / or information on the position of the sub-block (cu_sbt_pos_flag), restores residual signals by inversely transforming the transformation coefficients of the corresponding sub-block from the frequency domain to the spatial domain, and creates a final residual block for the current block by filling the areas that have not been inversely transformed with "0" values ​​of residual signals.

[0102] Additionally, when MTS is applied, the inverse transformation unit (530) determines a transformation function or transformation matrix to be applied in the horizontal and vertical directions, respectively, using MTS information (mts_idx) signaled from the video encoding device, and performs an inverse transformation on the transformation coefficients within the transformation block in the horizontal and vertical directions using the determined transformation function.

[0103] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is an intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is an inter prediction.

[0104] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0105] The inter prediction unit (544) determines the motion vector of the current block and the reference picture that the motion vector refers to using the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510), and predicts the current block using the motion vector and the reference picture.

[0106] The adder (550) restores the current block by adding the residual block output from the inverse transformation unit (530) and the prediction block output from the inter prediction unit (544) or the intra prediction unit (542). The pixels within the restored current block are used as reference pixels when intra-predicting the block to be decoded later.

[0107] The loop filter section (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as an in-loop filter. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-unit decoding. The SAO filter (564) and the ALF (566) perform additional filtering on the restored blocks after deblocking filtering to compensate for the difference between the restored pixels and the original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about the filter coefficients decoded from the non-stream.

[0108] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture to inter-predict blocks within the picture to be encoded later.

[0109]

[0110] An improved coding tool performed by the aforementioned image encoding device or image decoding device is disclosed below.

[0111]

[0112]

[0113] FIG. 6 is a diagram illustrating a neural network-based intra prediction method according to an embodiment of the present disclosure. Here, a neural network-based intra prediction block may refer to a prediction block generated by neural network-based intra prediction.

[0114] Referring to Fig. 6, the neural network-based intra prediction method may be a method of inputting the context (X) for the current block into a neural network model and performing intra prediction for the current block using the output value of the neural network model.

[0115] The context (X) can be generated using previously restored samples adjacent to the left and / or top of the current block (601). The generated context can be preprocessed. The preprocessed context ( ) can be input into a neural network model for predicting blocks.

[0116] A neural network-based intra prediction method can use a neural network model corresponding to the size of the current block. Here, the neural network model can be one of models of sizes 4×4, 8×4, 16×4, 32×4, 8×8, 16×8, or 16×16. A neural network model corresponding to a block of size w × h It can be referred to as such. A neural network model may consist of a combination of one or more layers or techniques among Fully Connected (FC) layers, Convolutional layers, Pooling layers, activation functions, Dropout, Batch Normalization, and Batch Norm layers. Here, the activation functions may include LeakyReLU and ReLU.

[0117] For example, a neural network model may have a structure consisting of 3 FC layers and 2 LeakyReLUs. Alternatively, a neural network model may have a structure consisting of 4 FC layers and 3 LeakyReLUs. Alternatively, a neural network model may have an iterative structure of FC layers and activation functions. The structure of the neural network model may vary depending on the size of the block to be predicted.

[0118] Neural network model It may be a model that takes context derived from previously restored pixels adjacent to the current block (601) as input and outputs intra-prediction related information of the current block. Here, downsampling or upsampling may be applied to the previously restored pixels.

[0119] The neural network model for predicting the current block has an initial predicted value for the block ( ), kernel index of LFNST (Low-Frequency Non-Separable Transform) (grpIdx i ) and the intra prediction mode index (repIdx) can be output. Here, downsampling or upsampling may be applied to the initial prediction value. Here, the kernel index may be the kernel index of LFNST MTS (multiple transform set) or NSPT (non-separable primary transformation).

[0120] Here, kernel index (grpIdx i ) can indicate whether to transpose the linear transformation coefficients when lfnstIdx of the linear transformation coefficients generated by applying DCT in vertical and horizontal directions to the residuals derived from the neural network-based intra prediction result is i.

[0121] Here, the intra prediction mode index (repIdx) may have a value from 0 to 66 and may indicate one of the non-directional intra prediction mode and / or directional intra prediction mode supported by the video codec specification.

[0122] And, the prediction block (603) for the current block is the initial prediction value ( It can be generated by applying postprocessing to ).

[0123]

[0124] In addition, according to one embodiment of the present disclosure, a final intra prediction block for the current block can be generated by weighting an intra prediction block generated using a neural network-based intra prediction block and a separate intra prediction mode.

[0125] According to one embodiment, a final intra prediction block for the current block can be generated by weighting an intra prediction block based on a neural network-based intra prediction block and an intra prediction block based on an intra prediction mode derived by the DIMD method.

[0126] Here, the DIMD method can be performed based on the slope histogram of surrounding samples adjacent to the current block. According to the DIMD method, boundary detection filters such as Sobel filters, Roberts cross filters, Prewitt filters, Charr filters, and Laplacian filters can be applied to surrounding samples adjacent to the current block. As a result of applying boundary detection filters to the surrounding samples of the current block, the slope of the restored sample can be calculated. Then, a slope histogram can be generated based on the calculated slope.

[0127] According to one embodiment, the intra prediction mode derived by the DIMD method may be an intra prediction mode mapped to the slope having the largest value among the slope histograms. For example, intra prediction modes mapped to the two slopes having the largest values ​​among the slope histograms may be used for DIMD-based intra prediction for the current block. Additionally, an undirected mode may be used for DIMD-based intra prediction for the current block.

[0128] Therefore, the intra-prediction block of the current block based on DIMD can be generated by weighting the intra-prediction block using an intra-prediction mode mapped to a slope selected from the histogram of slopes and the intra-prediction block using an undirected mode.

[0129] And, by combining the prediction samples of the intra-prediction block of the current block and the neural network-based intra-prediction block, the final intra-prediction block of the current block can be generated.

[0130]

[0131] According to one embodiment, the final intra-prediction block of the current block may be generated by combining prediction samples of an intra-prediction block using an intra-prediction mode derived from a histogram of gradients and a neural network-based intra-prediction block. For example, when generating the intra-prediction block of the current block based on DIMD, prediction samples of a neural network-based intra-prediction block may be used instead of an intra-prediction block using an undirected mode.

[0132] Alternatively, when generating an intra-prediction block of the current block based on DIMD, prediction samples from an intra-prediction block using an undirected mode or a neural network-based intra-prediction block may be selectively used. For example, the template cost of an intra-prediction block using an undirected mode and the template cost of a neural network-based intra-prediction block may be derived. Then, an intra-prediction block of the current block may be generated using the intra-prediction block having the lower template cost among the intra-prediction block using an undirected mode and the neural network-based intra-prediction block.

[0133] Alternatively, an intra prediction block of the current block can be generated by combining prediction samples from an intra prediction block using an intra prediction mode mapped to a selected gradient from a gradient histogram, an intra prediction block using an undirected mode, and a neural network-based intra prediction block.

[0134]

[0135] According to another embodiment, a final intra prediction block for the current block can be generated by weighting an intra prediction block based on an intra prediction mode derived by a neural network-based intra prediction block and an intra prediction block derived by a template-based intra prediction mode derivation (TIMD) method. The template-based intra prediction mode derivation method is described below.

[0136]

[0137] FIG. 7 is a drawing for explaining a template-based intra-prediction mode induction method according to one embodiment of the present disclosure.

[0138] Referring to FIG. 7, a prediction sample of the current template (720) can be generated by applying any candidate intra prediction modes to the reference pixels of the template (730), which are pixels adjacent to the current template (720), which is the template of the current block (710). Here, any candidate intra prediction mode may be an angular mode and / or a non-angular mode. The size of the current template (720) may be L1xN and / or L2xM, where L1, L2, N, and M are all positive integers.

[0139] Additionally, similarity between the predicted samples of the current template (720) and the samples reconstructed in the current template (720) can be derived. Similarity can be derived based on the cost value of the predicted samples of the current template (720) calculated using an arbitrary cost function. Here, among any candidate intra prediction modes, two intra prediction modes having the lowest cost value can be selected. Predicted samples of the current block (710) can be generated using the two selected intra prediction modes. Furthermore, the predicted samples of the current block (710) generated using the two selected intra prediction modes can be weighted summed using specific weight values. Here, a position dependent prediction combination (PDPC) can be applied to the predicted samples of the current block (710) generated using the two intra prediction modes. According to one embodiment, any candidate intra prediction modes used in the TIMD method may include MPM (most probable mode) and / or WAIP (wide angle intra prediction) modes.

[0140] According to one embodiment of the present disclosure, in an intra prediction based on the TIMD method, additional intra prediction candidate modes as well as MPM and / or WAIP modes may be used. That is, the candidate intra prediction modes used in the template-based intra prediction mode derivation method may be extended.

[0141] According to one embodiment, the final intra-prediction block of the current block may be generated by combining prediction samples of an intra-prediction block using an intra-prediction mode derived by the TIMD method and a neural network-based intra-prediction block. For example, in generating the intra-prediction block of the current block based on TIMD, prediction samples of a neural network-based intra-prediction block may be used instead of an intra-prediction block using an undirected mode.

[0142] Alternatively, when generating an intra-prediction block of the current block based on the TIMD method, prediction samples from an intra-prediction block using an undirected mode or a neural network-based intra-prediction block may be selectively used. For example, the template cost of an intra-prediction block using an undirected mode and the template cost of a neural network-based intra-prediction block may be derived. Then, an intra-prediction block of the current block may be generated using the intra-prediction block having the lower template cost among the intra-prediction block using an undirected mode and the neural network-based intra-prediction block.

[0143] Alternatively, an intra prediction block of the current block can be generated by combining prediction samples from an intra prediction block using an intra prediction mode induced by the TIMD method, an intra prediction block using a non-directed mode, and a neural network-based intra prediction block.

[0144]

[0145] According to another embodiment, a final intra prediction block for the current block can be generated by weighting the intra prediction block generated by the neural network-based intra prediction block and the intra prediction block generated by the intra template matching prediction method. Below, a template-based intra template matching prediction method is described.

[0146]

[0147] FIG. 8 is a drawing for explaining an intra-template matching mode prediction method according to one embodiment of the present disclosure.

[0148] Referring to FIG. 8, neighboring L-shaped areas (i.e., left, top, and top-left areas) of the current block (810) can be defined as the current template (820). And, within the predefined search range (R1, R2, R3, R4) of the reconstructed area (830) of the current picture (800), the reference template (840) most similar to the current template (820) can be searched.

[0149] The prediction block of the current block (810) can be derived based on the corresponding matching block (850) of the determined reference template (840). And, the block vector (860) is a vector indicating the matching block (850) of the current block in intra-template matching prediction.

[0150] In Fig. 8, the search ranges R1, R2, R3, and R4, which are defined in advance, can be defined as the current CTU (Coding Tree Unit) containing the current block, the top-left CTU, the top CTU, and the left CTU, respectively.

[0151] Additionally, reference templates can be explored within a predefined search range based on a predefined search order. For example, reference templates can be explored in a zigzag order of R1, R4, R3, and R2. Here, the search unit of the reference templates can be performed in pixel units or sub-pixel units. Furthermore, areas of partially overlapping reference templates can be clustered.

[0152] Meanwhile, information regarding the search range and the size and shape of the current template can be determined by the encoder and transmitted to the decoder. Additionally, the search range and the size and shape of the current template can be set to pre-defined values ​​in the encoder / decoder.

[0153] According to one embodiment, a block vector indicating a block within a search range may be searched. Alternatively, according to another embodiment, a block vector of a block adjacent to or non-adjacent to the current block may be derived as the block vector of the current block. According to yet another embodiment, the block vector may be an auto-relocated block vector prediction (AR-BVP) block vector.

[0154] In addition, the intra-predicted block of the current block can be generated based on the matching block indicated by the block vector, which is the intra-template matching prediction result. Furthermore, the final intra-predicted block of the current block can be generated by combining the prediction samples of the intra-predicted block of the current block and the neural network-based intra-predicted block.

[0155]

[0156] According to another embodiment, in intra prediction based on a spatial geometric partitioning mode, a final intra prediction block for the current block can be generated using a neural network-based intra prediction block.

[0157] According to the spatial geometric partitioning mode (SGPM), a coding unit block can be divided into two partitions of arbitrary shape according to a partitioning boundary. The partitioning boundary can be defined by various partitioning angles and offsets. Then, two prediction blocks can be generated by performing intra prediction on each of the two partitioned partitions according to different intra prediction modes. Here, the intra prediction mode for generating the two prediction blocks can be derived from a candidate list containing multiple intra prediction modes. The intra prediction modes in the candidate list can be rearranged based on template costs. Then, the final prediction block of the coding unit block is determined by weighted summing the two prediction blocks.

[0158] Here, neural network-based intra-prediction can be applied to at least one of the two partitions. For example, a prediction block for one partition may be generated using a directional prediction mode or a non-directional prediction mode, while a prediction block for the other partition may be generated using a neural network-based intra-prediction mode. Alternatively, a prediction block for both partitions may be generated using a neural network-based intra-prediction mode. Or, a final intra-prediction block for the current block may be generated by weighting the prediction blocks for the two partitions generated using a directional prediction mode and a non-directional prediction mode, and the neural network-based intra-prediction block.

[0159]

[0160] According to another embodiment, a final intra prediction block for the current block can be generated by weighting the intra prediction blocks generated by a neural network-based intra prediction block and a template-based multi-reference line (TMRL) intra prediction method. The template-based multi-reference line intra prediction method is described below.

[0161] FIG. 9 is a diagram illustrating a template-based multi-reference sample line intra prediction method according to one embodiment of the present disclosure.

[0162] Referring to FIG. 9, TMRL intra prediction can be performed using an intra prediction mode for the current block (910) and N multiple reference samples. The number of intra prediction mode candidates used in TMRL intra prediction may be M. Here, M is a positive integer greater than or equal to 1. Among the intra prediction modes, any mode preset in the decoder may not be used in TMRL intra prediction. According to one embodiment, the preset arbitrary mode may be a planar mode or an undirected prediction mode.

[0163] Also, referring to FIG. 9, multiple sample lines (920, 931, 932, 933) used for TMRL intra prediction are shown. Here, the nearest neighbor sample line (reference line 0, 920) to the current block (910) is used as a template, and the next N adjacent reference sample lines (931, 932, 933) can be used as multiple reference sample lines. In this case, N is a positive integer greater than or equal to 2.

[0164] There are a total of N x M combinations of N reference sample lines (931, 932, 933) and M intra-frame prediction modes. And, for TMRL intra prediction, a candidate list can be generated that includes candidates indicating different combinations of intra prediction modes and reference sample lines used for intra prediction.

[0165] According to one embodiment, the candidate list for TMRL intra prediction may use all N x M combinations. On the other hand, according to another embodiment, the candidate list for TMRL intra prediction may include only K candidates determined based on the cost value of the intra prediction value based on the combination of the intra prediction mode and the reference sample line. Here, the cost value may be the result of a cost function based on the comparison result between the prediction value using the template derived by template matching and the intra prediction value using the combination of the intra prediction mode and the reference sample line.

[0166] Here, the K candidates may be selected from N x M combinations in order of smallest cost value. The K candidates may be sorted in a fixed order (e.g., sorted in ascending order based on cost value) to form a candidate list. Here, K can be an integer greater than or equal to 1 and less than or equal to N x M.

[0167] Here, the cost function may use at least one of the sum of absolute differences (SAD), sum of squared differences (SSD), and sum of absolute transformed differences (SATD).

[0168] According to one embodiment, the final intra-prediction block of the current block can be generated by combining the prediction samples of an intra-prediction block based on a combination of a determined intra-prediction mode and a reference sample line and a neural network-based intra-prediction block.

[0169] In addition to the embodiments described above, a final intra prediction block for the current block can be generated by combining an intra prediction block for the current block using various intra prediction modes and a neural network-based intra prediction block.

[0170]

[0171] Cost values ​​for each prediction mode can be calculated to select one of the intra prediction modes and the neural network-based intra prediction mode. Here, the cost values ​​can be calculated through various cost calculation methods. Specifically, the cost values ​​can be calculated using a single cost calculation method determined from among various cost calculation methods or by combining various cost calculation methods. The cost calculation methods may be methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), SATD (Sum of Absolute Transformed Difference), and MR-SAD (Mean Reduced Sum of Absolute Difference).

[0172] The error cost for each prediction mode can be calculated based on the sample value of the current template adjacent to the current block. For example, the predicted value of the current template can be generated using the samples adjacent to the current template and the prediction mode. Additionally, the error cost for the prediction mode can be calculated based on the difference between the predicted value of the current template and the previously restored value of the current template.

[0173] To calculate cost values, one or more cost estimation methods may be used. For example, cost values ​​may be calculated by combining the SATD and SAD methods, combining the SATD and MR-SAD methods, or combining the SATD, SAD, and MR-SAD methods. When using two or more cost estimation methods, the final cost value may be calculated based on the average or weighted sum of cost values ​​according to the different cost estimation methods.

[0174]

[0175] To perform a weighted sum of prediction blocks based on different intra-prediction modes and neural network-based intra-prediction blocks, weights for each prediction block can be set. Here, the weights can be set through various methods.

[0176] For example, the weights applied to different prediction blocks can all be set equally. That is, the final prediction value can be generated by simply averaging the different prediction blocks. Alternatively, the weights applied to different prediction blocks can be set to fixed values. Or, the weights applied to different prediction blocks can be set based on the coding parameters of the current block. For example, coding parameters can be set by referring to a predefined table based on slice type (I / P / B), signal components (Y / U / V), QP, block size, shape, etc. Alternatively, the weights applied to different prediction blocks can be set based on the error cost of each prediction block. The weight of a prediction block with a lower error cost can be set higher. In other words, the weight of each prediction block can be set inversely proportional to the error cost of each prediction block.

[0177]

[0178] Prior to weighted summing the prediction blocks based on different intra-prediction modes and the neural network-based intra-prediction blocks, the samples of the prediction blocks may be corrected. The samples of the prediction blocks may be corrected through at least one of bilateral filtering, boundary smoothing, PDPC, and gradient PDPC methods. Additionally, the correction of the prediction block samples may be performed optionally.

[0179]

[0180] According to one embodiment of the present disclosure, a prediction block of a current block may be generated by combining a neural network-based intra prediction block for the current block and an inter prediction block for the current block.

[0181] For example, if the prediction mode of the current block is CIIP mode, the prediction block of the current block may be generated by weighted summing the intra prediction block of the current block and the inter prediction block of the current block. The intra prediction block of the current block may be a prediction block generated by a neural network-based intra prediction method instead of a predefined Planar mode or an intra prediction mode derived by the TIMD method. Alternatively, the intra prediction block of the current block may be a prediction block generated by a mode selected from Planar mode, an intra prediction mode derived by the TIMD method, or a neural network-based intra prediction mode. Alternatively, the intra prediction block of the current block may be a block generated through the weighted summing of a prediction block generated by Planar mode and a prediction block generated by a neural network-based intra prediction method.

[0182] Alternatively, if the prediction mode of the current block is a geometrical partitioning mode (GPM), the prediction block of the current block can be generated by weighting the intra prediction block for the first partition partitioned from the current block and the inter prediction block for the second partition partitioned from the current block. The geometrical partitioning mode is described below.

[0183] FIG. 10 is a diagram illustrating a combination between inter-prediction and intra-prediction that may occur in a geometric division mode according to one embodiment of the present disclosure.

[0184] Referring to Fig. 10, two partitions divided according to the geometric partitioning mode can each be independently predicted by intra-prediction or inter-prediction.

[0185] A combination of intra prediction and inter prediction may be applied to blocks (1001) and (1002) of FIG. 10. According to the prediction combination, one of the two partitions is predicted according to intra prediction, and the other is predicted according to inter prediction.

[0186] A prediction mode that generates prediction blocks for two regions using intra prediction and inter prediction can be referred to as a geometric partitioning mode based on intra / inter prediction (GPM with inter / intra prediction).

[0187] Here, the intra prediction block of the current block may be a prediction block generated by a neural network-based intra prediction method. Alternatively, the intra prediction block of the current block may be a block generated based on one of the prediction modes included in the intra prediction candidate mode list. Here, the intra prediction candidate mode list may include a neural network-based intra prediction mode and intra prediction modes.

[0188]

[0189] According to another embodiment, a final intra prediction block for the current block can be generated by weighting the intra prediction block generated by the neural network-based intra prediction block and the intra prediction block generated by the intra block copy prediction method. The intra block copy prediction method is described below.

[0190] FIG. 11 is a diagram illustrating an intra-block copy-based prediction method according to one embodiment of the present disclosure.

[0191] Referring to FIG. 11, a matching block (1140) can be searched within a predefined search range (R1, R2, R3, R4) of the reconstructed area (1130) of the current picture (1100) based on the block vector (1120) of the current block (1110). Then, a predicted block of the current block (1110) can be derived based on the matching block (1140). Thus, when an intra-block copy-based prediction method is applied to the current block (1110), the encoder can transmit information regarding the intra-block copy-based prediction method and information related to the block vector (1120) to the decoder.

[0192] R1, R2, R3, and R4, which are predefined search ranges in FIG. 11, can be defined as the current CTU (Coding Tree Unit) containing the current block (1110), the upper-left CTU located at the top left of the current CTU within an area within a predefined range from the current CTU, the upper CTU located at the top of the current CTU within an area within a predefined range from the current CTU, and the left CTU located to the left of the current CTU within an area within a predefined range from the current CTU, respectively. Here, the area within the predefined range may be an area set as one of the current tile, slice, or picture.

[0193] In addition, the final intra prediction block of the current block can be generated by weighting the intra prediction block generated by the neural network-based intra prediction block and the intra prediction block generated by the intra block copy prediction method.

[0194] According to another embodiment, the final intra-prediction block of the current block can be generated using a first prediction block induced by an intra-merge mode and a second prediction block generated by a neural network-based intra-prediction method.

[0195] According to the intra-merge mode, a candidate list containing multiple intra-prediction candidates may be constructed based on the intra-prediction modes of neighboring blocks that are adjacent to or not adjacent to the current block. For example, the intra-prediction modes of neighboring blocks may be one of the following prediction modes: directional intra-prediction mode, non-directional intra-prediction mode, DIMD, OBIC, intra-merge mode, TIMD, MPM-based mode, intra-TMP, etc. Then, based on an index indicating one or more candidates from the intra-prediction mode candidate list, the intra-prediction mode of the current block may be set, and a prediction block for the current block may be generated based on the set intra-prediction mode.

[0196] According to one embodiment, a first prediction block is generated based on one intra prediction candidate included in a candidate list, and a second prediction block can be generated by a neural network-based intra prediction method.

[0197] Meanwhile, according to another embodiment, the intra prediction mode candidate list may further include a neural network-based intra prediction mode. The neural network-based intra prediction mode may be added to the existing intra prediction mode candidate list or may replace one of the candidates in the candidate list. In this case, the prediction mode for generating the first prediction block may be set based on an index indicating any candidate among the candidate list. And, the prediction mode for generating the second prediction block may be set based on an index indicating a neural network-based intra prediction mode among the candidate list.

[0198] And, the first prediction block and the second prediction block can be combined in a weighted sum manner based on at least one cost value or weight to generate the final intra prediction block of the current block.

[0199] According to another embodiment, the final prediction block for the color difference component of the current block can be generated using a first color difference prediction block generated based on a color difference component intra prediction mode and a second color difference prediction block generated by neural network-based intra prediction.

[0200] For example, a color difference component intra-prediction mode may include a color difference intra-prediction mode that utilizes the correlation between a luminance component and a color difference component, a direct prediction mode for the color difference component, or a prediction mode that selects at least one of a plurality of color difference prediction block candidates or combines different color difference prediction block candidates.

[0201] The second color difference prediction block can be generated by a neural network model that takes as input adjacent reference samples of the color difference component, luminance information corresponding to the color difference component, or a combination thereof.

[0202] In addition, the first color difference prediction block and the second color difference prediction block may be used selectively with each other or combined based on at least one cost value or weight to generate a final color difference prediction block.

[0203]

[0204] According to one embodiment of the present disclosure, the neural network-based intra prediction may be a neural network-based merge intra prediction that utilizes information of surrounding blocks adjacent to the current block.

[0205] Specifically, when one or more neighboring blocks adjacent to the current block are intra-predicted using a neural network, the neural network-based merge intra-prediction may be a mode that performs intra-prediction based on a merge candidate list containing neighboring blocks. Here, the merge candidate may be a block located within a certain range of distance from the current block. Additionally, the merge candidate list may include information related to intra-prediction of two or more neighboring blocks. For the neural network-based prediction of the current block, the predicted value of the neighboring block according to the neural network-based prediction mode, the intra-prediction mode information used in the prediction process of the neighboring block, and / or the intra-prediction mode information used in the transformation process for the neighboring block may be used. Furthermore, the final predicted block of the current block may be generated by weighting the predicted block of the current block generated using the merge candidate list and the predicted block of the current block generated by the neural network-based intra-prediction method.

[0206]

[0207] According to one embodiment of the present disclosure, a neural network-based intra prediction may be a method for generating a final prediction block for a current block by inputting sample values ​​of reference regions having different shapes into a neural network and then summing the different prediction values ​​generated. The reference region may be a region containing previously restored samples adjacent to the current block. Additionally, the size of the reference region may be adaptively set.

[0208] Below, embodiments of reference regions having different forms according to one embodiment of the present disclosure are described.

[0209]

[0210] FIGS. 12A to 12C are drawings for illustrating candidates for the shape of a reference region according to one embodiment of the present disclosure.

[0211] Referring to FIGS. 12A to 12C, the shape of the reference region may have one of a plurality of reference region shape candidates.

[0212] Here, the reference region may include surrounding samples on the left and top of the block as illustrated in FIG. 12A. A candidate reference region shape including surrounding samples on the left and top of the block may be referred to as an L-shaped reference region. The size of the L-shaped reference region can be expressed as (w x L2) + (L1 x h). Here, w and h represent the width and height of the block, and the values ​​of L1 and L2 may be any positive numbers.

[0213] Alternatively, the reference region may include a peripheral sample to the left of the block as illustrated in FIG. 12B. A candidate reference region shape including a peripheral sample to the left of the block may be referred to as the left reference region. The size of the left reference region may be denoted by (L3 x h). Here, h represents the height of the block, and the value of L3 may be any positive number.

[0214] Alternatively, the reference region may include a peripheral sample at the top of the block as illustrated in FIG. 12C. A candidate reference region shape including a peripheral sample at the top of the block may be referred to as the upper reference region. The size of the upper reference region may be represented as (w x L4). Here, w represents the width of the block, and the value of L4 may be any positive number.

[0215] Alternatively, the reference region may include restoration samples that are not adjacent to the current block.

[0216]

[0217] That is, according to one embodiment of the present disclosure, the final prediction block for the current block can be generated by weighting different prediction values ​​generated by inputting sample values ​​of reference regions having different shapes into a neural network.

[0218] In addition, to perform a weighted sum of the prediction blocks, weights can be set for each prediction block. Here, the weights can be set through various methods.

[0219] For example, the weights applied to different prediction blocks can all be set equally. That is, the final prediction value can be generated by simply averaging the different prediction blocks. Alternatively, the weights applied to different prediction blocks can be set to fixed values. Or, the weights applied to different prediction blocks can be set based on the coding parameters of the current block. For example, coding parameters can be set by referring to a predefined table based on slice type (I / P / B), signal components (Y / U / V), QP, block size, shape, etc. Alternatively, the weights applied to different prediction blocks can be set based on the error cost of each prediction block. The weight of a prediction block with a lower error cost can be set higher. In other words, the weight of each prediction block can be set inversely proportional to the error cost of each prediction block.

[0220] The error cost for each prediction block of the current block can be calculated based on the sample value of the current template adjacent to the current block. For example, the predicted value of the current template can be generated using samples adjacent to the current template and the prediction mode. Furthermore, the error cost for the prediction mode can be calculated based on the difference between the predicted value of the current template and the previously restored value of the current template.

[0221] Here, the shape of the current template may be identical to the shape of the reference region used for neural network-based prediction. For example, if samples from an L-shaped reference region are used to generate a neural network-based intra-prediction block, the error cost of the prediction block may be calculated based on the error between the predicted value for the L-shaped template and the reconstructed value of the L-shaped template. Alternatively, if samples from a left-side reference region are used to generate a neural network-based intra-prediction block, the error cost of the prediction block may be calculated based on the error between the predicted value for the left-side template and the reconstructed value of the left-side template. Alternatively, if samples from an upper-side reference region are used to generate a neural network-based intra-prediction block, the error cost of the prediction block may be calculated based on the error between the predicted value for the upper-side template and the reconstructed value of the upper-side template. Alternatively, if reference samples not adjacent to the current block are used to generate a neural network-based intra-prediction block, the error cost of the prediction block may be calculated based on the error between the predicted value and the reconstructed value for the region containing the reference samples not adjacent to the current block.

[0222] Here, the error cost value can be calculated through various cost calculation methods. Specifically, the cost value can be calculated using a single cost calculation method determined from among various cost calculation methods, or by combining various cost calculation methods. The cost calculation methods may be methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), SATD (Sum of Absolute Transformed Difference), and MR-SAD (Mean Reduced Sum of Absolute Difference).

[0223]

[0224] FIG. 13 is a flowchart illustrating an image decoding method according to one embodiment of the present disclosure. Specifically, FIG. 13 is a flowchart for explaining a method for generating a prediction block of a current block by weighted summing a prediction block generated based on a neural network-based intra prediction method and a prediction block generated based on a separate prediction mode in an image decoding method. The intra prediction method of FIG. 13 can be performed by a decoding device.

[0225] The image decoder can induce a prediction mode of the current block (S1310). Based on the prediction mode of the current block, the image decoder can generate a first prediction block of the current block (S1320).

[0226] According to one embodiment, the prediction mode of the current block may be an intra prediction mode derived by the DIMD (Decoder side Intra Mode Derivation) method. And, the first prediction block may be generated by combining prediction blocks generated based on each of the plurality of intra prediction modes derived by the DIMD method.

[0227] According to another embodiment, the prediction mode of the current block may be an intra prediction mode derived by the Template-based Intra Mode Derivation (TIMD) method. And, the first prediction block may be generated by combining prediction blocks generated based on each of the plurality of intra prediction modes derived by the TIMD method.

[0228] According to another embodiment, the prediction mode of the current block may be a geometric partition mode. Here, the first prediction block may be a prediction block for a first partition partitioned from the current block, and the second prediction block may be a prediction block for a second partition partitioned from the current block.

[0229] According to another embodiment, the prediction mode of the current block may be an inter-prediction mode. And, the first prediction block may be a prediction block generated as an inter-prediction result for the current block.

[0230] According to another embodiment, the prediction mode of the current block may be an intra-template matching prediction mode or an intra-block copy mode. And, the first prediction block may be a prediction block generated based on a block indicated by the block vector of the current block.

[0231] According to another embodiment, the prediction mode of the current block may be a template-based multi-reference sample line intra-prediction mode. And, the first prediction block may be generated based on a combination of a reference sample line and an intra-prediction mode selected from a combination of a plurality of reference sample lines and intra-prediction modes.

[0232] According to another embodiment, the prediction mode of the current block may be a neural network-based merge intra prediction mode. And, the first prediction block may be generated based on information derived by inputting the context of a neighboring block adjacent to the current block into a neural network model.

[0233] According to another embodiment, the prediction mode of the current block may be a neural network-based intra-prediction mode. The first prediction block may be generated by inputting a context derived based on reference samples of a region adjacent to the current block into a neural network model. Here, the region adjacent to the current block may be determined as one of a first candidate reference region including samples adjacent to the left of the current block, a second candidate reference region including samples adjacent to the top of the current block, and a third candidate reference region including samples adjacent to the left of the current block and samples adjacent to the top of the current block.

[0234] The image decoder can derive the context of the current block based on a reference sample adjacent to the current block (S1330).

[0235] The image decoding device can input the context of the current block into a neural network model to generate a second prediction block of the current block (S1340).

[0236] The image decoding device can generate a final prediction block of the current block based on the weighted sum of the first prediction block and the second prediction block (S1350). Specifically, the final prediction block of the current block can be generated by applying a first weight and a second weight to the first prediction block and the second prediction block, respectively. Here, the first weight and the second weight can be derived based on the distortion value between the predicted value and the restored value of the template area of ​​the first prediction block and the distortion value between the predicted value and the restored value of the template area of ​​the second prediction block. Furthermore, the template area of ​​the first prediction block may correspond to an area adjacent to the current block.

[0237]

[0238] The steps described in FIG. 13 can be performed in the same way in a video encoding method. Additionally, a bitstream can be generated by a video encoding method including the steps described in FIG. 13. The bitstream can be stored on a non-transient computer-readable recording medium and can also be transmitted (or streamed).

[0239]

[0240] Although the flowcharts and timing diagrams in this specification describe each process as being executed sequentially, this is merely an illustrative explanation of the technical concept of one embodiment of the present disclosure. In other words, a person skilled in the art to which one embodiment of the present disclosure belongs may modify and adapt the flowcharts and timing diagrams in various ways, such as changing the order described in the flowcharts and timing diagrams or executing one or more of the processes in parallel, without departing from the essential characteristics of one embodiment of the present disclosure; therefore, the flowcharts and timing diagrams are not limited to a chronological order.

[0241] It should be understood that the exemplary embodiments described above may be implemented in many different ways. The functions or methods described in one or more examples may be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as “...unit” to particularly emphasize their implementation independence.

[0242] Meanwhile, the various functions or methods described in the present embodiment may be implemented as instructions stored in a non-transient recording medium that can be read and executed by one or more processors. A non-transient recording medium includes, for example, any type of recording device in which data is stored in a form readable by a computer system. For example, a non-transient recording medium includes storage media such as an EPROM (erasable programmable read-only memory), a flash drive, an optical drive, a magnetic hard drive, and a solid-state drive (SSD).

[0243] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment.

[0244] The present invention can be used in a device for encoding / decoding images and a recording medium storing a bitstream.

Claims

1. In a video decoding method, Step for inducing the prediction mode of the current block; A step of generating a first prediction block of the current block based on the prediction mode of the current block; A step of deriving the context of the current block based on a reference sample adjacent to the current block; A step of inputting the context of the current block into a neural network model to generate a second predicted block of the current block; and A video decoding method comprising the step of generating a final prediction block of the current block based on the weighted sum of the first prediction block and the second prediction block.

2. In Paragraph 1, The prediction mode of the current block above is, An image decoding method that is an intra prediction mode derived by the DIMD (Decoder side Intra Mode Derivation) method.

3. In Paragraph 2, The above first prediction block is, An image decoding method generated by combining prediction blocks generated based on each of a plurality of intra prediction modes derived by the above DIMD method.

4. In Paragraph 1, The prediction mode of the current block above is, An image decoding method that is an intra prediction mode derived by the TIMD (Template-based Intra Mode Derivation) method.

5. In Paragraph 4, The above first prediction block is, An image decoding method generated by combining prediction blocks generated based on each of a plurality of intra prediction modes derived by the above TIMD method.

6. In Paragraph 1, The above first prediction block is, An image decoding method, wherein the prediction block is generated based on the block indicated by the block vector of the current block.

7. In Paragraph 1, The above first prediction block is, An image decoding method generated based on a combination of a reference sample line and an intra prediction mode selected from a combination of multiple reference sample lines and intra prediction modes.

8. In Paragraph 1, The above first prediction block is, An image decoding method generated by performing inter-prediction on the above current block.

9. In Paragraph 1, The above first prediction block is, It is a prediction block for the first partition divided from the above current block, and The above second prediction block is, Image decoding method, which is a prediction block for a second partition divided from the above current block.

10. In Paragraph 1, The above first prediction block is, An image decoding method generated based on information derived by inputting the context of surrounding blocks adjacent to the current block into a neural network model.

11. In Paragraph 1, The above first prediction block is, An image decoding method generated by inputting a context derived based on reference samples of the region adjacent to the current block into the above neural network model.

12. In Paragraph 11, The final predicted block of the current block mentioned above is, It is generated by applying a first weight and a second weight to the first prediction block and the second prediction block, respectively, and The above first weight and second weight are, An image decoding method characterized by being derived based on a distortion value between a predicted value and a restored value in a template area of ​​the first prediction block and a distortion value between a predicted value and a restored value in a template area of ​​the second prediction block.

13. In Paragraph 1, The area adjacent to the current block mentioned above is, It is determined to be one of a first candidate reference region including samples adjacent to the left of the current block, a second candidate reference region including samples adjacent to the top of the current block, and a third candidate reference region including samples adjacent to the left of the current block and samples adjacent to the top of the current block. The template area of ​​the first prediction block above is, An image decoding method characterized by corresponding to an area adjacent to the current block.

14. In a video encoding method, Step for inducing the prediction mode of the current block; A step of generating a first prediction block of the current block based on the prediction mode of the current block; A step of deriving the context of the current block based on a reference sample adjacent to the current block; A step of inputting the context of the current block into a neural network model to generate a second predicted block of the current block; and An image encoding method comprising the step of generating a final prediction block of the current block based on the weighted sum of the first prediction block and the second prediction block.

15. A bitstream transmission method generated by a video encoding method, The above transmission method includes the step of transmitting the bitstream, and The above encoding method is, Step for inducing the prediction mode of the current block; A step of generating a first prediction block of the current block based on the prediction mode of the current block; A step of deriving the context of the current block based on a reference sample adjacent to the current block; A step of inputting the context of the current block into a neural network model to generate a second predicted block of the current block; and A transmission method comprising the step of generating a final prediction block of the current block based on the weighted sum of the first prediction block and the second prediction block.