Sub-block-based inter prediction method, and video encoding and decoding apparatus performing same

Abstract

The present invention relates to a sub-block-based inter prediction method, and a video encoding and decoding apparatus performing same. The method comprises the steps of: obtaining motion information of neighboring sub-blocks spatially adjacent to a current block; deriving motion information of sub-blocks in the current block by using the obtained motion information of the neighboring sub-blocks; and adding a sub-block-based spatial motion vector prediction candidate to a candidate list for a sub-block-based merge mode by using the derived motion information of the sub-blocks.

Description

Sub-block based inter-prediction method, image encoding and decoding device performing the same

[0001] The present invention relates to a sub-block-based inter-prediction method that enables efficient representation of motion information.

[0002] Compression encoding of video signals is achieved by removing redundant information by considering spatial correlations, temporal correlations, probabilistic correlations, etc.

[0003] Among the prediction methods used in video encoding, intra prediction is a method that predicts the current block by referring to already reconstructed samples around the block to be encoded.

[0004] Recently, due to advancements in various media and data transmission methods, user demand for high-resolution, high-quality video is increasing.

[0005] As encoded data for high-resolution images contains a larger amount of information than encoded data for low- or medium-resolution images, the cost of transmitting or storing it increases.

[0006] To address this problem, research on encoding and decoding methods to effectively reduce the amount of encoded data for high-resolution images is continuing.

[0007] The present invention aims to provide a sub-block-based inter-prediction method capable of improving image compression performance by efficiently representing motion information, and an image encoding and decoding device for performing the same.

[0008] An inter prediction method according to an embodiment of the present invention is a method for performing inter prediction based on sub-blocks, comprising: a step of obtaining motion information of surrounding sub-blocks spatially adjacent to a current block; a step of deriving motion information of sub-blocks within the current block using the obtained motion information of surrounding sub-blocks; and a step of adding a sub-block-based spatial motion vector prediction (sbSMVP) candidate to a candidate list for a sub-block-based merge mode using the derivation motion information of sub-blocks.

[0009] sbSMVP candidates corresponding to one or more available directions among the multiple directions from which movement information is derived from the surrounding sub-blocks are added to the candidate list.

[0010] Meanwhile, if at least one of the surrounding sub-blocks required to induce movement information according to the above direction does not have movement information, the sbSMVP type of the direction may be set to not-available.

[0011] The image encoding device according to one embodiment of the present invention performs the inter-prediction method as described above.

[0012] The image decoding device according to one embodiment of the present invention performs the inter prediction method as described above.

[0013] Meanwhile, at least some of the steps of the above methods may be implemented in a computer-readable recording medium that records a program for execution on a computer, and may be provided as the program itself.

[0014] According to one embodiment of the present invention, by deriving motion vectors in sub-block units, motion vectors can be expressed more precisely, thereby improving the compression performance of the image.

[0015] Figure 1 is a block diagram illustrating the configuration and operation of a video encoder for encoding images.

[0016] FIG. 2 is a diagram illustrating an example of a method for dividing blocks of an image.

[0017] Figure 3 is a block diagram illustrating the configuration and operation of a video decoder for decoding an image.

[0018] FIG. 4a is a diagram illustrating an example of surrounding blocks spatially adjacent to the current block.

[0019] Figure 4b is a diagram illustrating an example of a method for scaling motion vectors.

[0020] FIG. 5 is a diagram illustrating embodiments of surrounding blocks spatially adjacent to the current block.

[0021] FIGS. 6 and FIGS. 7 are drawings for illustrating embodiments of a method for scaling motion vectors.

[0022] FIGS. 8 and 9 are drawings for illustrating embodiments of a method for deriving sub-block-based spatial motion vector prediction (sbSMVP) candidates.

[0023] Hereinafter, an image encoding and decoding method and apparatus according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0024] In the following description of the present invention, specific descriptions of related known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, their definitions should be based on the overall content of the present invention.

[0025] In addition, the preferred embodiments of the present invention described below will focus on explaining the functional configurations that must be additionally provided for the present invention, while omitting as much as possible the system functional configurations that are already provided in each system functional configuration or are ordinarily provided in the technical field to which the present invention belongs, in order to efficiently explain the technical components constituting the present invention.

[0026] If a person skilled in the art to which the present invention pertains can easily understand the function of a component that has been used in the past among the functional configurations that are omitted and not illustrated below, and can also clearly understand the relationship between the component that was omitted as above and the component added for the present invention.

[0027] In this specification, a device that encodes an image to generate a video signal bitstream is referred to as an encoding device, an encoding device, or an encoder, and a device that decodes a video signal bitstream to restore an image is referred to as a decoding device, a decoding device, or a decoder.

[0028] A pixel or pel refers to the smallest unit that constitutes an image, and the terms pixel and sample may be used interchangeably. Generally, a sample can represent a pixel or its value, and it may represent only the pixel or its value of the luminance component, or only the pixel or its value of the chroma component.

[0029] Furthermore, the term "unit" is used to refer to a basic unit of image processing or a specific location within a picture, representing an image region containing at least one of the luminance component and the chrominance component. Specifically, the term "unit" can be used as a concept encompassing the Coding Tree Unit (CTU), Coding Unit (CU), Prediction Unit (PU), and Transform Unit (TU). Additionally, the term "block" represents an image region containing a specific component among the luminance and chrominance components, and an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. Here, terms such as unit, block, partition, signal, and region may be used interchangeably.

[0030] Meanwhile, the term "picture" refers to a field or a frame, and these terms may be used interchangeably. For example, if the image is interlaced, a single frame is separated into an odd (or top) field and an even (or bottom) field, and each field is configured as a single picture unit for encoding or decoding. And if the image is sequential, a single frame is configured as a picture for encoding or decoding.

[0031]

[0032] FIG. 1 is a block diagram illustrating an encoding device according to an embodiment of the present invention, and is intended to explain the configuration and operation of a video encoder for encoding images.

[0033] Referring to FIG. 1, the video encoder (100) may be configured to include a conversion unit (110), a quantization unit (120), an inverse quantization unit (130), an inverse conversion unit (140), a filtering unit (150), a prediction unit (160), a DPB (Decoded Picture Buffer, 170), and an entropy coding unit (180).

[0034] The conversion unit (110) converts the residual signal, which is the difference between the input video signal and the prediction signal generated by the prediction unit (160), to obtain a conversion coefficient value.

[0035] For example, the Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), or Wavelet Transform may be used.

[0036] The transformation kernel used for the transformation of a residual block may be a transformation kernel having separable characteristics of vertical transformation and horizontal transformation. In this case, the transformation of the residual block can be performed by separating it into a vertical transformation and a horizontal transformation. For example, an encoder may perform a vertical transformation by applying a transformation kernel in the vertical direction of the residual block. Additionally, an encoder may perform a horizontal transformation by applying a transformation kernel in the horizontal direction of the residual block.

[0037] Meanwhile, a transformation kernel can be used as a term referring to a set of parameters used for transforming a residual signal, such as a transformation matrix, transformation array, transformation function, or transformation; it can be any one of a plurality of available kernels, and transformation kernels based on different transformation types may be used for vertical transformation and horizontal transformation, respectively.

[0038] Transformation coefficients may be distributed such that higher coefficients are found towards the top-left corner of the block, and coefficients close to '0' are found towards the bottom-right corner. Additionally, as the current block size increases, there is a higher likelihood of '0' coefficients existing in the bottom-right region; to reduce the transformation complexity of large blocks, only an arbitrary top-left region may be retained, while the remaining regions may be reset to '0'.

[0039] In addition, an error signal may exist only in some areas of a coding block, in which case the conversion process may be performed only on some arbitrary areas. For example, in a block of size 2Nx2N, an error signal may exist only in the first 2NxN block, in which case the conversion process is performed only on the first 2NxN block, but the second 2NxN block may not be encoded or decoded without the conversion process being performed.

[0040] The encoder may perform additional transformations before the transformation coefficients are quantized. The transformation method described above is referred to as a primary transform, and the additional transformation may be referred to as a secondary transform.

[0041] The second conversion can be optional for each residual block, and for example, the encoder can improve coding efficiency by performing a second conversion for areas where it is difficult to concentrate energy in the low-frequency region with only the first conversion.

[0042] Specifically, a second transformation may be additionally performed on blocks where residual values ​​appear significantly in directions other than the horizontal or vertical direction of the residual block, and unlike the first transformation, the second transformation may not be performed separately into a vertical transformation and a horizontal transformation. The second transformation described above may be referred to as a Low Frequency Non-Separable Transform (LFNST).

[0043] The quantization unit (120) quantizes the conversion coefficient value output from the conversion unit (110).

[0044] In order to increase coding efficiency, instead of coding the picture signal as is, a method is used to predict the picture using an area that has already been coded through a prediction unit (160), and to obtain a restored picture by adding the residual value between the original picture and the predicted picture to the predicted picture.

[0045] To prevent mismatches from occurring between the encoder and decoder, when performing prediction in the encoder, information available in the decoder must be used; to this end, the encoder may perform a process of restoring the currently encoded block.

[0046] The inverse quantization unit (130) inversely quantizes the conversion coefficient value, and the inverse conversion unit (140) restores the residual value using the inversely quantized conversion coefficient value.

[0047] The filtering unit (150) performs filtering operations using a deblocking filter, a Sample Adaptive Offset (SAO), an Adaptive Loop Filter (ALF), etc., to improve the quality of the restored picture and enhance encoding efficiency.

[0048] A deblocking filter is a filter for removing distortion within blocks generated at the boundaries between blocks in a restored picture, and the encoder can determine whether to apply the deblocking filter to a boundary based on the distribution of pixels included in a few columns or rows relative to an arbitrary boundary within a block.

[0049] When a deblocking filter is applied, the filtering unit (150) can apply a long filter, a strong filter, or a weak filter depending on the deblocking filtering strength, and can process horizontal filtering and vertical filtering in parallel.

[0050] Sample Adaptive Offset (SAO) can be used to correct the offset from the original image on a pixel-by-pixel basis for a residual block to which a deblocking filter has been applied. The filtering unit (150) may use a Band Offset method to correct the offset for a specific picture by dividing the pixels included in the image into a certain number of regions, determining the region to perform offset correction, and applying the offset to the region. Additionally, the filtering unit (150) may use an Edge Offset method to apply the offset by considering the edge information of each pixel.

[0051] An Adaptive Loop Filter (ALF) is a method that divides pixels contained in an image into specific groups, determines a single filter to apply to each group, and performs differential filtering for each group. Information regarding whether to apply an Adaptive Loop Filter can be signaled at the coding unit level, and the shape and filter coefficients of the ALF filter to be applied may vary depending on each block. Additionally, the same type of Adaptive Loop Filter may be applied regardless of the characteristics of the current block to be applied.

[0052] The filtered picture can be stored in DPB (170) to be used as a reference picture.

[0053] The prediction unit (160) includes an intra / IBC prediction unit (161) and an inter prediction unit (165). The intra / IBC prediction unit (161) performs intra prediction and IBC prediction within the current picture, and the inter prediction unit (165) performs inter prediction by predicting the current picture using a reference picture stored in the DPB (170).

[0054] The intra / IBC prediction unit (161) performs intra prediction from the restored regions within the current picture and transmits intra encoding information to the entropy coding unit (180). Here, the intra encoding information may include at least one of an intra prediction mode, an MPM (Most Probable Mode) flag, an MPM index, and information regarding a reference sample.

[0055] Additionally, the intra / IBC prediction unit (161) can perform IBC (Intra Block Copy) prediction from the restored samples in the current picture and transmit the IBC encoding information to the entropy coding unit (180).

[0056] In this case, the intra / IBC prediction unit (161) obtains a block vector value indicating a reference area used for prediction of the current area by referring to a specific area within the current picture, and can perform IBC prediction using the obtained block vector value.

[0057] The intra / IBC prediction unit (161) can transmit IBC encoding information, including at least one of the size information of the reference area and block vector information (index information for predicting the block vector of the current block within the motion candidate list and block vector difference information), to the entropy coding unit (160).

[0058] The inter prediction unit (165) refers to a specific area of ​​the restored reference picture to find the part most similar to the current area and obtains a Motion Vector (MV) value, which is the distance between the areas, and transmits motion information for the obtained reference area (reference direction indicator information (L0 prediction, L1 prediction, bidirectional prediction), reference picture index, motion vector information, etc.) to the entropy coding unit (180).

[0059] Additionally, the inter prediction unit (165) performs motion compensation using motion information to generate a prediction block for the current block, and transmits inter encoding information containing motion information for a reference area to the entropy coding unit (180).

[0060] Meanwhile, quantized transformation coefficients in the form of a two-dimensional array can be rearranged into a one-dimensional array for entropy coding.

[0061] The method for scanning quantized transform coefficients can be determined by the size of the transform block and the intra-prediction mode, and diagonal, vertical, or horizontal scans can be applied, and the scan information can be signaled in block units or derived from a decoder according to a defined rule.

[0062] The entropy coding unit (180) generates a bitstream by entropy coding information representing quantized conversion coefficients, intra-coding information, and inter-coding information, etc. For this purpose, a variable length coding (VLC) method and an arithmetic coding method may be used.

[0063] Variable Length Coding (VLC) converts input symbols into a sequence of codewords, the length of which can be variable. For example, frequently occurring symbols can be represented by short codewords, while infrequently occurring symbols can be represented by long codewords.

[0064] As a variable-length coding method, Context-based Adaptive Variable Length Coding (CAVLC) can be used.

[0065] Arithmetic coding utilizes the probability distribution of each data symbol to convert consecutive data symbols into a single prime number, thereby obtaining the optimal prime bit required to represent each symbol.

[0066] As an arithmetic coding method, the Context-based Adaptive Binary Arithmetic Code (CABAC) method can be used.

[0067] CABAC is a method of binary arithmetic encoding using multiple context models generated based on probabilities obtained through experiments. A context model can also be referred to as a context model. First, if a symbol is not in binary form, the encoder binarizes each symbol using exp-Golomb or similar methods. Binarized 0s or 1s can be described as bins.

[0068] The CABAC initialization process is divided into context initialization and arithmetic coding initialization. Context initialization is the process of initializing the occurrence probability of each symbol, which is determined by the type of symbol, quantization parameters (QP), and slice type (whether it is I, P, or B). A context model containing this initialization information can use probability-based values ​​obtained through experiments.

[0069] The context model provides the probability of occurrence of LPS (Least Probable Symbol) or MPS (Most Probable Symbol) for the symbol currently to be coded, and information (valMPS) on which empty value between 0 and 1 corresponds to the MPS.

[0070] One of several context models is selected through the context index (ctxIdx), and the context index can be derived from information about the block currently to be encoded or information about surrounding blocks.

[0071] Initialization for binary arithmetic coding is performed based on the probability model selected from the context model. Binary arithmetic coding proceeds by dividing into probability intervals based on the occurrence probabilities of 0 and 1, and then the probability interval corresponding to the bin to be processed becomes the entire probability interval for the next bin to be processed.

[0072] The location information within the probability interval where the last bin was processed is output. However, since the probability interval cannot be divided indefinitely, if it shrinks to within a certain size, a renormalization process is performed to widen the probability interval and output the corresponding location information. Additionally, after each bin is processed, a probability update process may be performed to newly set the probability for the next bin to be processed based on the information of the processed bin.

[0073] As mentioned above, the generated bitstream is encapsulated with a Network Abstraction Layer (NAL) unit as the basic unit.

[0074] NAL units are classified into VCL (Video Coding Layer) NAL units containing video data and non-VCL NAL units containing parameter information for decoding video data, and various types of VCL or non-VCL NAL units may exist.

[0075] A NAL unit consists of NAL header information and data, which is a Raw Byte Sequence Payload (RBSP), and the NAL header information includes summary information about the RBSP. The RBSP of a VCL NAL unit contains encoded integer coding tree units.

[0076] In order to decode a bitstream in a decoder, the bitstream must first be separated into NAL units, and then each separated NAL unit must be decoded. Meanwhile, the information required for decoding the bitstream can be transmitted in the Picture Parameter Set (PPS), Sequence Parameter Set (SPS), Video Parameter Set (VPS), etc.

[0077] Meanwhile, the configuration and operation of the encoder described with reference to FIG. 1 are according to an embodiment of the present invention, and some configurations may be omitted or added as needed.

[0078]

[0079] Additionally, a single picture can be divided and encoded into sub-pictures, slices, tiles, etc. A sub-picture may include one or more slices or tiles. If a single picture is divided and encoded into multiple slices or tiles, it can be displayed on the screen only after all slices or tiles within the picture have been fully decoded.

[0080] When a single picture is encoded into multiple sub-pictures, any single sub-picture may be decoded and displayed on the screen. A slice may include multiple tiles or sub-pictures, and a tile may include multiple sub-pictures or slices.

[0081] Sub-pictures, slices, and tiles can be encoded or decoded independently of each other, which is effective for parallel processing and improving processing speed; however, since encoded information from adjacent sub-pictures, slices, and tiles cannot be utilized, the bit size may increase.

[0082] And sub-pictures, slices, and tiles can be divided into multiple Coding Tree Units (CTUs) and encoded.

[0083] The coding tree unit may consist of a 128x128 luminance coding tree block (Coding Tree Block, CTB) and two 64x64 color difference coding tree blocks.

[0084] A single coding tree unit may not be divided and may constitute a single coding unit (CU) itself, or it may be divided into multiple coding units as shown in FIG. 2. A coding unit may consist of a luminance coding block (CB) and two color difference coding blocks.

[0085] A single coding unit may consist of a single transform unit (TU) or be divided into multiple transform units. A transform unit may consist of a luminance transform block (TB) and two chrominance transform blocks.

[0086] Here, a coding unit represents a basic unit for processing a picture during the prediction, transformation, quantization, entropy coding, and decoding processes, and the size and shape of the coding unit within a single picture may not be constant.

[0087] The coding unit may have a square or non-square shape, and the non-square coding unit may include a vertical coding unit in which the height is greater than the width and a horizontal coding unit in which the width is greater than the height.

[0088] The coding tree unit is first partitioned into a Quad Tree (QT) structure, so that a single node of size 2NX2N can be partitioned into four nodes of size NXN. Meanwhile, the Quad Tree partitioning can be performed recursively, and not all nodes need to be partitioned to the same depth.

[0089] The leaf nodes of a quad tree can be further divided into a Multi-Type Tree (MTT) structure. For example, in a Multi-Type Tree structure, a single node can be divided into a binary or ternary tree structure of horizontal or vertical splitting. Accordingly, there can be four types of splitting structures in a Multi-Type Tree structure: vertical binary splitting, horizontal binary splitting, vertical ternary splitting, and horizontal ternary splitting.

[0090] In each tree structure, the width and height of the nodes can both have powers of 2. For example, in a binary tree (BT) structure, a node of size 2NX2N can be divided into 2 NX2N nodes by a vertical binary split and into 2 2NXN nodes by a horizontal binary split.

[0091] In addition, in a ternary tree (TT) structure, a node of size 2NX2N can be divided into (N / 2)X2N, NX2N, and (N / 2)X2N nodes by vertical ternary partitioning, and into 2NX(N / 2), 2NXN, and 2NX(N / 2) nodes by horizontal ternary partitioning. This multi-type tree partitioning can be performed recursively.

[0092] Leaf nodes of a multi-type tree can serve as coding units, and if a coding unit is not larger than the maximum transformation length, it can be used as a unit of prediction and transformation without further splitting. On the other hand, if the width or height of a coding unit is greater than the maximum transformation length, it can be split into multiple transformation units without explicit signaling regarding splitting.

[0093] The tree partitioning structure described above may have the same form (Single Tree) for the luminance block and the chrominance block, or different forms (Dual Tree) for the luminance block and the chrominance block.

[0094] Meanwhile, regarding the block division from the coding tree unit (CTU) to the coding unit (CU) as described above, the final coding unit can be determined by selecting the division structure with the smallest rate-distortion cost (RD cost) value within the allowable size and depth conditions through the rate-distortion optimization (RDO) process.

[0095] In addition, the syntax for expressing block partitions as described above may be as shown in Table 1 below.

[0096]

[0097] In Table 1, split_cu_flag indicates whether block partitioning has occurred or partitioning has ended, split_qt_flag indicates whether QT partitioning has occurred, mtt_split_cu_vertical_flag indicates whether MTT partitioning is vertical or horizontal, and mtt_split_cu_binary_flag indicates whether MTT partitioning is binary or ternary.

[0098]

[0099] FIG. 3 is a block diagram illustrating a decoding device according to an embodiment of the present invention, and is intended to explain the configuration and operation of a video decoder for encoding images.

[0100] Referring to FIG. 3, the video decoder (200) may be configured to include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transform unit (230), a filtering unit (240), a prediction unit (260), and a DPB (Decoded Picture Buffer, 270).

[0101] The entropy decoding unit (210) entropies decodes the bitstream to extract conversion coefficient information, intra-coding information, inter-coding information, etc. for each region.

[0102] For example, the entropy decoding unit (210) can obtain a binary code for conversion coefficient information of a specific region from the bitstream, and can obtain quantized conversion coefficients by debinding the binary code.

[0103] The inverse quantization unit (220) inversely quantizes the quantized conversion coefficients, and the inverse conversion unit (230) restores the residual value using the inversely quantized conversion coefficients.

[0104] Meanwhile, the residual value obtained from the inverse conversion unit (230) is added to the predicted value obtained from the prediction unit (260) to restore the original pixel value.

[0105] The filtering unit (240) performs filtering operations using a deblocking filter, a sample adaptive offset, an adaptive loop filter, etc., to improve the quality of the restored picture, and the filtered picture can be stored in the DPB (270) to be output or used as a reference picture for the next picture.

[0106] The prediction unit (260) includes an intra / IBC prediction unit (261) and an inter prediction unit (265), and generates a prediction picture by utilizing the encoding type decoded through the entropy decoding unit (210), the conversion coefficient for each region, intra / inter encoding information, etc.

[0107] To restore the current block being decrypted, the decrypted regions of the current picture containing the current block or other pictures may be utilized. A picture (or tile / slice) that performs intra prediction or intra-BC prediction using only the current picture for restoration is called an intra picture or I picture (or tile / slice), and a picture (or tile / slice) capable of performing intra prediction, inter prediction, and intra-BC prediction is called an inter picture (or tile / slice).

[0108] Meanwhile, a picture (or tile / slice) that uses at most one motion vector and reference picture index to predict sample values ​​of each block among the inter-pictures (or tiles / slices) is called a predictive picture or P picture (or tile / slice), and a picture (or tile / slice) that uses at most two motion vectors and reference picture indices is called a bi-predictive picture or B picture (or tile / slice).

[0109] That is, P picture (or tile / slice) uses at most one set of motion information to predict each block, and B picture (or tile / slice) uses at most two sets of motion information to predict each block. Here, a set of motion information may include one or more motion vectors and one reference picture index.

[0110] The intra / IBC prediction unit (261) generates a prediction block using intra-coding information and restored samples within the current picture, and the intra-coding information may include at least one of an intra-prediction mode, an MPM (Most Probable Mode) flag, and an MPM index.

[0111] The intra / IBC prediction unit (261) can predict the sample values ​​of the current block by using the restored samples located to the left and / or above the current block as reference samples.

[0112] For example, the reference samples may be samples adjacent to the left boundary of the current block and / or samples adjacent to the upper boundary, and among the samples of the surrounding blocks of the current block, samples located on a line within a preset distance from the left boundary of the current block and / or samples located on a line within a preset distance from the upper boundary of the current block. In this case, the surrounding blocks of the current block may include at least one of the left (L) block, upper (A) block, lower-left (BL) block, upper-right (AR) block, or upper-left (AL) block adjacent to the current block.

[0113] Additionally, the intra / IBC prediction unit (261) can restore the current region by referring to a specific region containing restored samples within the current picture. To this end, the intra / IBC prediction unit (261) can perform IBC prediction using IBC encoding information obtained from the entropy decoding unit (210), and the IBC encoding information may include block vector information.

[0114] The inter prediction unit (265) generates a prediction block using reference picture and inter encoding information stored in the DPB (270), and the inter encoding information may include a set of motion information of the current block for the reference block (reference picture index, motion vector, etc.).

[0115] Meanwhile, inter-prediction can include L0 prediction, L1 prediction, and bi-prediction.

[0116] L0 prediction means a prediction using one reference picture included in the L0 picture list, and L1 prediction means a prediction using one reference picture included in the L1 picture list. To do this, a set of motion information (e.g., motion vectors and reference picture indices, etc.) may be required.

[0117] In the pair prediction method, up to two reference regions can be used, and the two reference regions may exist in the same reference picture or in different pictures. Accordingly, in the up to two sets of motion information used in the pair prediction method, two motion vectors may correspond to the same reference picture index or to different reference picture indices.

[0118] At this time, the reference pictures are pictures located temporally before or after the current picture, and can be completed pictures that have already been restored, and the two reference regions used in the pair prediction method can be regions selected from the L0 picture list and the L1 picture list, respectively.

[0119] The inter prediction unit (265) can obtain a reference block of the current block using a motion vector and a reference picture index, and the reference block exists within a reference picture corresponding to the reference picture index.

[0120] In addition, a sample value of a block specified by a motion vector or an interpolated value thereof can be used as a predictor for the current block. For motion prediction with subpel-level pixel accuracy, an 8-tap interpolation filter can be used for the luminance signal and a 4-tap interpolation filter can be used for the chrominance signal.

[0121] Meanwhile, the configuration and operation of the decoder described with reference to FIG. 3 is according to an embodiment of the present invention, and some configurations may be omitted or added as needed, and the decoder can decode an image by performing the reverse process of the encoding method of the encoder described above.

[0122]

[0123] Hereinafter, the inter prediction method according to the present invention will be described in detail.

[0124] Inter prediction modes may include Skip / Merge modes and AMVP (Advanced Motion Vector Prediction) modes.

[0125] In skip / merge mode, a candidate list is constructed using movement information of spatially or temporally adjacent neighboring blocks, and the movement information of the current block can be represented using the index value of the selected movement information within the candidate list.

[0126] Meanwhile, merge modes can be divided into regular merge and sub-block merge modes.

[0127] In general merge mode, a merge candidate list is configured, and movement compensation can be performed by using the movement information of a specific candidate pointed to by an index among the merge candidate list as the movement information of the corresponding block (CU, Coding Unit).

[0128] Specifically, the merge candidate list may be composed of Spatial Merge Candidates, Temporal Motion Vector Prediction (TMVP) candidates, History-based Motion Vector Prediction (HMVP) candidates, Pairwise Average Candidates, and Zero Motion Vector (Zero MV).

[0129] Here, Temporal Motion Vector Prediction (TMVP) is a technique that derives motion information from collocated blocks within a reference picture existing at a different temporal location from the current picture and adds it to a merge candidate list or AMVP candidate list.

[0130] In Temporal Motion Vector Prediction (TMVP), a collocated picture, which is one of the reference pictures, is determined, and a collocated block within the collocated picture can be determined.

[0131] For example, information about collocated pictures and information about the predicted direction can be determined on a slice basis and included in a slice header as shown in Table 2 below.

[0132]

[0133] Referring to Table 2, the prediction direction information is specified by the 'sh_collocated_from_l0_flag' syntax of the slice header, and the index of the same location picture can be specified by the 'sh_collocated_ref_idx' syntax.

[0134] For example, a picture at the same location can be determined as the reference picture with index "2" among the reference pictures in reference picture list 0 (L0) when 'sh_collocated_from_l0_flag' is "1" and 'sh_collocated_ref_idx' is "2", and as the reference picture with index "3" among the reference pictures in reference picture list 1 (L1) when 'sh_collocated_from_l0_flag' is "0" and 'sh_collocated_ref_idx' is "3".

[0135] Meanwhile, the collocated block can be determined as a surrounding block (BR) located at the bottom-right relative to the block at the same location as the current block within the collocated picture, as shown in FIG. 4a.

[0136] Here, if the surrounding block (BR) at the bottom right is not available for reasons such as being located outside the picture boundary, or if it is encoded using an intra-prediction method, the block (CT) corresponding to the center position relative to the block at the same position as the current block within the same position picture may be determined as the same position block.

[0137] And in the step of scaling the motion vector, the motion vector of the same position block can be converted into the motion vector corresponding to the current block.

[0138] Here, when a TMVP candidate is added to the merge candidate list, the reference picture of the current block is set to the same location picture, and when a TMVP candidate is added to the AMVP candidate list, the reference picture of the current block can be explicitly signaled.

[0139] Referring to (a) of Fig. 4b, after scaling based on the Picture Order Count (POC) difference (curr_Poc_diff), which is the distance between the current picture (curr_pic) and the same-location picture (col_pic), and the POC difference (col_Poc_diff), which is the distance between the same-location picture (col_pic) and the reference picture (col_ref_pic) of the same-location block (col_blk), it can be included in the merge candidate list as a TMVP candidate for the current block (curr_blk).

[0140] For example, the scaling factor applied to the motion vector of the same location block (col_blk) can be calculated as the ratio between two POC difference values ​​(curr_Poc_diff, col_Poc_diff) as shown in Equation 1 below.

[0141]

[0142] Meanwhile, referring to (b) of FIG. 4b, the current block (curr_blk) can be included in the AMVP candidate list as a temporal TMVP candidate after scaling based on the difference in Picture Order Count (POC) (curr_Poc_diff) between the current picture (curr_pic) and the reference picture (curr_ref_pic) of the current block (curr_blk), and the difference in POC between the same location picture (col_pic) and the same location block (col_blk) (col_ref_pic).

[0143] For example, the scaling factor applied to the motion vector of the same location block (col_blk) can be calculated as the ratio between two POC difference values ​​(curr_Poc_diff, col_Poc_diff) as shown in Equation 2 below.

[0144]

[0145] Meanwhile, in addition to the general merge mode described above, MMVD (Merge with Motion Vector Difference) mode, CIIP (Combined Inter and Intra Prediction) mode, and TRIANGULAR mode can be used as inter prediction modes.

[0146] In sub-block based merge mode, the current block can be divided into multiple sub-blocks of a certain size, motion vectors can be calculated on a sub-block basis, and motion compensation for the corresponding sub-block can be performed using the motion vectors of each sub-block.

[0147] Meanwhile, the sub-block-based merge candidate list may include sub-block Temporal Motion Vector Prediction (sbTMVP) candidates and affine merge candidates with an affine motion model applied.

[0148] If the candidate index value for the motion information of the current block refers to the sbTMVP candidate among the sub-block-based merge candidate lists, a temporal motion vector can be derived at the sub-block level and motion compensation can be performed.

[0149] In sub-block-based temporal motion vector prediction (sbTMVP), a collocated block within a collocated picture is divided into multiple sub-blocks, and a motion vector for each sub-block of the collocated block can be derived.

[0150] After that, the motion vectors derived for each sub-block of the same location block are mapped to values ​​corresponding to the reference picture of the current block and can be used as motion vectors for each sub-block of the current block.

[0151] Specifically, the sub-block-based temporal motion vector prediction (sbTMVP) method may sequentially include the steps of determining the location of blocks at the same location, obtaining representative motion information of blocks at the same location, obtaining motion information for each sub-block, and scaling motion vectors.

[0152] First, in the step of determining the location of a block at the same location, the location indicated by the movement information of a surrounding block adjacent to the current block or the same location within the picture at the same location can be determined as the location of the block at the same location.

[0153] Here, information about the same location picture may be included in the slice header as described above with reference to Table 2.

[0154] For example, referring to FIG. 5, by using the movement information of the block (A1) located on the left among the surrounding blocks of the current block, the block at the position indicated by the corresponding movement vector within the same position picture can be determined as the same position block.

[0155] Specifically, to determine the same location block, the prediction mode of the surrounding block (A1) of the current block is checked, and it can be determined whether the surrounding block (A1) is encoded in an inter-prediction mode.

[0156] If the surrounding block (A1) is encoded in an inter-prediction mode and the reference picture of the reference picture list 0 (L0) for the surrounding block (A1) is the same as the same location picture of the current block (first condition), then the block at the location pointed to by the motion vector in the L0 direction for the surrounding block (A1) within the same location picture can be determined as the same location block.

[0157] Meanwhile, if the above first condition is not satisfied, the surrounding block (A1) is encoded in an inter-prediction mode, and if the reference picture of the reference picture list 1 (L1) for the surrounding block (A1) is the same as the same location picture of the current block (second condition), the block at the location pointed to by the movement vector in the L1 direction for the surrounding block (A1) within the same location picture can be determined as the same location block.

[0158] And if the above first and second conditions are not satisfied, a block at the same location as the current block within the same location picture may be determined as the same location block.

[0159] Next, in the step of obtaining representative motion information of a block at the same location, if a collocated sub-block corresponding to a central position within the block at the same location is encoded in an inter-prediction mode, the motion vector of the collocated sub-block at the same location can be set as the representative motion vector (base MV) of the block at the same location.

[0160] Specifically, the prediction mode of a sub-block at the same location corresponding to the central location within the block at the same location is identified, and in this case, the sub-block at the same location may be a block encoded in an inter-prediction mode of not only unidirectional but also bidirectional.

[0161] If the corresponding same-location sub-block is encoded in an inter-prediction mode and motion information for the corresponding same-location sub-block exists, the sbTMVP mode is available, and the motion vector of the corresponding same-location sub-block can be set as the representative motion vector (base MV) of the same-location block.

[0162] Meanwhile, if the same location sub-block is not encoded in the inter-prediction mode and there is no movement information for the same location sub-block, the sbTMVP mode may not be available.

[0163] Next, in the step of obtaining movement information for each sub-block, the movement vector of each sub-block within the same location block can be set to a movement vector corresponding to the center position of the corresponding sub-block.

[0164] Here, the motion vector corresponding to the central position of the sub-block may mean a motion vector within a motion information storage (e.g., MV memory) that covers the central sample position of the sub-block.

[0165] Specifically, the prediction mode of each sub-block within the same location block can be checked first.

[0166] If the corresponding sub-block is encoded in an inter-prediction mode and motion information for the corresponding sub-block exists, a motion vector in the MV memory covering the central sample position of the corresponding sub-block can be set as the motion vector of the corresponding sub-block.

[0167] Meanwhile, if the sub-block is not encoded in an inter-prediction mode and there is no motion information for the sub-block, the representative motion vector (base MV) of the same location block can be set as the motion vector of the sub-block.

[0168] And in the step of scaling motion vectors, the motion vectors of each sub-block within the same location block can be converted into motion vectors corresponding to the current block.

[0169] Referring to Fig. 6(a), the position of a block (col_blk) in the same position within a picture (col_pic) is determined by using movement information for a surrounding block (nb_blk) of a current block (curr_blk) within a current picture (curr_pic), and movement information of each sub-block within a block (col_blk) in the same position can be obtained.

[0170] After that, the motion vector of each sub-block within the same-location block (col_blk) can be scaled using the POC difference (curr_Poc_diff) between the current picture (curr_pic) and the reference picture (curr_ref_pic) of the current block (curr_blk), and the POC difference (col_Poc_diff) between the same-location picture (col_pic) and the reference picture (col_ref_pic) of the same-location block (col_blk).

[0171] For example, the scaling factor applied to the motion vector of each sub-block within the same location block (col_blk) can be calculated as the ratio between two POC difference values ​​(curr_Poc_diff, col_Poc_diff), as shown in Equation 2 above.

[0172] Referring to Fig. 6(b), since the sub-block-based temporal motion vector prediction (sbTMVP) is added to the sub-block-based merge candidate list, when the reference picture (curr_ref_pic) of the current block (curr_blk) is set to the same location picture (col_pic), the scaling factor applied to the motion vector of each sub-block within the same location block (col_blk) may be calculated as the ratio between the two POC difference values ​​(curr_Poc_diff, col_Poc_diff) as in Equation 1 above.

[0173] Meanwhile, in the sub-block-based temporal motion vector prediction (sbTMVP) method described above, the size of the sub-block may be the same as the size unit in which motion information of the blocks is stored in a motion information storage (e.g., MV memory or buffer) to be used for future temporal motion vector prediction (TMVP).

[0174] For example, if motion information is stored in MV memory in units of 8x8, the size of the sub-block for sbTMVP may be 8x8, and if motion information is stored in MV memory in units of 4x4, the size of the sub-block for sbTMVP may be 4x4.

[0175] Additionally, the reference picture (curr_ref_pic) of the current block (curr_blk) for sbTMVP may be the picture corresponding to the 0th index of the reference picture list indicated by the syntax 'sh_collocated_from_l0_flag' in the slice header.

[0176] Referring to FIG. 7, the surrounding block (A1) of the current block in the current picture is encoded in an inter-prediction mode, and when the reference picture of the surrounding block (A1) is the same as the collocated picture for the current picture, the block at the position pointed to by the motion vector of the surrounding block (A1) in the collocated picture is set as the collocated block, and motion vector shifting (MV shifting) from the position of the current block in the current picture to the collocated block in the collocated picture can occur based on the movement of the surrounding block (A1).

[0177] In addition, blocks at the same location are divided into multiple sub-blocks, and the central location of blocks at the same location may be a location indicated by a dot.

[0178] Meanwhile, motion information within the MV memory covering the central location sample of the same location block can be set as the representative motion vector (base MV) for the same location block.

[0179] And when setting motion vectors for each sub-block within the same location block, for sub-blocks where motion information covering the central location sample of each sub-block is not in the MV memory, a representative motion vector (base MV) can be set as the motion vector of that sub-block.

[0180] The motion vector of each sub-block within the same location block is mapped (MV mapping) to correspond to the reference picture of the current block to be decoded, and can be used as the motion vector for each sub-block within the current block.

[0181] AMVP mode is an inter-prediction mode that transmits the Motion Vector Difference (MVD), which is the difference between the motion vector of the current block and the Motion Vector Prediction Value (MVP) derived from surrounding blocks.

[0182] Here, the AMVP Candidate List may be composed of Spatial Merge Candidates, Temporal Motion Vector Prediction (TMVP) Candidates, History-based Motion Vector Prediction (HMVP) Candidates, and Zero Motion Vector (Zero MV).

[0183] Meanwhile, in addition to the regular AMVP mode described above, the symmetric MVD mode and the AFFINE AMVP mode can be used as inter-prediction modes.

[0184] In addition, decoder-based motion vector correction techniques such as DMVR (Decoder-side MV Refinement), BDOF (Bi-Directional Optical Flow), and PROF (Prediction Refinement with Optical Flow) can be additionally applied to merge mode and AMVP mode.

[0185]

[0186] A sub-block-based inter-prediction method according to an embodiment of the present invention may include a process of deriving motion vector candidates from spatially and temporally adjacent neighboring blocks for each of the sub-blocks of a block to be decoded.

[0187] Specifically, a sub-block-based inter-prediction method according to an embodiment of the present invention may include: a step of obtaining motion information of surrounding sub-blocks spatially adjacent to a current block; a step of deriving motion information of sub-blocks within the current block using the obtained motion information of surrounding sub-blocks; and a step of adding a sub-block-based spatial motion vector prediction (sbSMVP) candidate to a candidate list for a sub-block-based merge mode using the derivation motion information of sub-blocks.

[0188] In this case, sbSMVP candidates corresponding to one or more available directions among the multiple directions from which movement information is derived from the surrounding sub-blocks may be added to the candidate list.

[0189] Here, for a surrounding sub-block among the surrounding sub-blocks that is not encoded in an inter-prediction mode, movement information of another sub-block adjacent to the surrounding sub-block or preset default movement information may be used.

[0190] Meanwhile, if at least one of the surrounding sub-blocks required to induce movement information according to the above direction does not have movement information, the sbSMVP type of the direction may be set to not-available.

[0191] In addition, if there is no movement information for a specific number or fewer of the surrounding sub-blocks required to induce movement information according to the above direction, the sbSMVP type of that direction can be set as available.

[0192] And the sub-block-based spatial motion vector prediction (sbSMVP) candidate, the sbSMVP candidate, and the affine merge candidate can be added to the candidate list in that order.

[0193] The method further includes the step of performing a redundancy check on sbSMVP candidates corresponding to each of the above available directions, and sbSMVP candidates with identical movement information for the sub-blocks may not be added redundantly to the candidate list.

[0194] Additionally, the method may further include the step of performing a re-ordering operation using Template Matching (TM) on sbSMVP candidates corresponding to each of the above-mentioned available directions; and / or the step of performing a refinement operation using TM on the movement information of the above-mentioned induced sub-blocks.

[0195] The above sub-block-based inter-prediction method can be performed by an image encoding device or an image decoding device according to an embodiment of the present invention.

[0196]

[0197] The sub-block-based inter prediction method described below may include at least some of the inter prediction methods described above with reference to FIGS. 1 to 7, and the description of the same as the inter prediction method described above will be omitted below.

[0198] According to one embodiment of the present invention, a sub-block-based merge candidate list may include motion vector candidates of the following four groups, and when configuring the candidate list, motion vector candidates of one or more of the following four groups may be used.

[0199] 1) Sub-block based spatial motion vector prediction (sbSMVP) candidates

[0200] 2) Sub-block based temporal motion vector prediction (sbTMVP) candidates

[0201] 3) Affine Merge Candidate

[0202] 4) GPM (Geometric Partitioning Mode) Merge Candidates

[0203] Hereinafter, embodiments of a method for deriving motion vector candidates from spatially and temporally adjacent neighboring blocks for each sub-block of the current block, which is the block to be decoded, are described.

[0204] However, the motion vector candidate derivation process at the sub-block level according to the embodiment of the present invention is not limited to the sub-block-based merge mode and can also be used when constructing a candidate list for the regular merge mode, MMVD (Merge with Motion Vector Difference), and / or AMVP mode.

[0205] Meanwhile, a group of sub-block based merge modes including a regular sub-block merge mode, an affine sub-block merge mode, and / or a GPM sub-block merge mode may be configured, and when configuring a candidate list for the regular sub-block merge mode among the group of sub-block based merge modes, sbSMVP candidates and / or sbTMVP candidates may be used.

[0206] And when constructing the candidate list by dividing it into sbSMVP candidates and sbTMVP candidates, the maximum number of each group is limited to k and h, so that when the candidate list is filled with sbSMVP candidates up to the maximum number k or sbTMVP candidates up to the maximum number h, no more candidates of that group may be added to the candidate list.

[0207] For example, the maximum number of sbSMVP candidates (k) can be set to 5, and the maximum number of sbTMVP candidates (h) can be set to 4.

[0208] As another example, the maximum number of all candidates (n) included in the candidate list can be set, and information about the maximum number (n) of the candidate list can be included as a High-Level Syntax (HLS) value in at least one of the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), and Slice Header (SH).

[0209] Meanwhile, during the process of constructing the candidate list, when the number of candidates added to the candidate list reaches the maximum number (n), the process of constructing the candidate list ends.

[0210] Up to k sbSMVP candidates may be derived and included in the candidate list for sub-block-based merge mode, for example, multiple sbSMVP candidates may be derived and up to 5 non-duplicate sbSMVP candidates may be added to the candidate list.

[0211] An embodiment of the present invention provides a method for deriving motion information from spatially adjacent surrounding blocks for each sub-block within a current block, which is a block to be decoded.

[0212] For example, referring to FIG. 8, a current block (curr_blk) of size 16x16 is divided into a sub-block (sub_blk) of size 4x4, and there may be a neighboring block (nb_sub_blk) spatially adjacent to the current block (curr_blk).

[0213] In the process of deriving sbSMVP candidates, the size of each sub-block (sub_blk) within the current block (curr_blk) and the surrounding block (nb_sub_blk) of the current block (curr_blk) may be equal to the unit size of the movement information storage (e.g., MV memory or buffer) where the movement information of the corresponding block is stored so that it can be used as movement information for other blocks in the future.

[0214] For example, if motion information is stored in MV memory with a size of 4x4, the sub-block (sub_blk) within the current block (curr_blk) and the surrounding block (nb_sub_blk) of the current block (curr_blk) can each have a size of 4x4.

[0215] Hereinafter, neighboring blocks spatially adjacent to the current block will be referred to as neighbor sub-blocks, to distinguish them from sub-blocks within the current block.

[0216] A number of surrounding sub-blocks equal to '(current block width / 4)×2' adjacent to the upper boundary of the current block, a number of surrounding sub-blocks equal to '(current block height / 4)×2' adjacent to the left boundary, and a surrounding sub-block on the upper left side are set, so that a total of '{(current block width / 4)×2 + (current block height / 4)×2 + 1}' surrounding sub-blocks can be set for the current block.

[0217] However, the location and number of surrounding sub-blocks as described above are according to an embodiment of the present invention, and the present invention is not limited thereto. One or more surrounding sub-blocks spatially adjacent to the current block may be set, and a non-adjacent spatial block that is not spatially adjacent to the current block may also be set as a surrounding sub-block.

[0218] Meanwhile, in the process of inducing movement information for sub-blocks within the current block from surrounding sub-blocks, there may be a directionality in which the movement information is induced.

[0219] For example, there may be five directions as illustrated in FIG. 9, but the present invention is not limited thereto, and movement information for each sub-block within the current block may be induced from surrounding sub-blocks according to one or more directions.

[0220] Referring to FIG. 9(a), movement information for sub-blocks can be induced according to the horizontal direction, for example, movement information of surrounding sub-blocks (11, 12, 13, 14) adjacent to the left boundary of the current block can be inherited in the horizontal direction to the sub-blocks within the current block.

[0221] Referring to Fig. 9(b), movement information for sub-blocks can be induced according to the vertical direction, for example, movement information of surrounding sub-blocks adjacent to the upper boundary of the current block can be inherited in the vertical direction to the sub-blocks within the current block.

[0222] Referring to Fig. 9 (c), movement information for sub-blocks can be induced according to the right-down diagonal direction, for example, movement information of adjacent sub-blocks adjacent to the upper-left side can be inherited by sub-blocks within the current block in the right-down diagonal direction.

[0223] Referring to Fig. 9 (d), movement information for sub-blocks can be induced according to the right-up diagonal direction, for example, movement information of adjacent sub-blocks in the lower left direction can be inherited by sub-blocks within the current block in the right-up diagonal direction.

[0224] Referring to Fig. 9 (e), movement information for sub-blocks can be induced according to the left-down diagonal direction, for example, movement information of adjacent sub-blocks adjacent to the upper right can be inherited by sub-blocks within the current block in the left-down diagonal direction.

[0225]

[0226] A method for deriving a candidate for a sub-block-based spatial motion vector prediction (sbSMVP) according to an embodiment of the present invention may include the steps of obtaining motion information of surrounding sub-blocks, deriving motion information of sub-blocks within the current block, and adding a candidate for sbSMVP to a candidate list of sub-block-based merge modes.

[0227] Below, examples for each step of the sbSMVP candidate induction method will be described in detail.

[0228]

[0229] In the step of obtaining movement information of surrounding sub-blocks, movement information of surrounding sub-blocks spatially adjacent to the current block, which is the block to be decoded, is obtained.

[0230] If the prediction mode of a surrounding sub-block is identified and the surrounding sub-block is encoded in an inter-prediction mode, the movement information of the surrounding sub-block may be available.

[0231] Here, the movement information of a surrounding sub-block may refer to movement information within an MV memory covering the upper-left (or center) position of the surrounding sub-block, but the present invention is not limited thereto.

[0232] In one embodiment according to the present invention, a peripheral sub-block encoded in an inter-prediction mode may be set to an available state, and a peripheral sub-block not encoded in an inter-prediction mode may be set to a not-available state.

[0233] In another embodiment, for a surrounding sub-block not encoded in an inter-prediction mode (e.g., a surrounding sub-block in an unavailable state), movement information of at least one of the surrounding sub-blocks located above, below, to the left, and to the right of the surrounding sub-block may be used as movement information of the surrounding sub-block.

[0234] For example, referring to FIG. 8, if there is no movement information for the surrounding sub-block at position 12, the movement information for the surrounding sub-block at position 11 above and / or the surrounding sub-block at position 13 below can be used as movement information for the surrounding sub-block at position 12.

[0235] In another embodiment, for a surrounding sub-block not encoded in an inter-prediction mode (e.g., a surrounding sub-block in an unavailable state), any default motion information may be used as motion information for the surrounding sub-block.

[0236] For example, referring to FIG. 8, for a surrounding sub-block at position 12 where no motion information exists, the 0th reference picture in reference picture list 0 (L0) is assigned as the reference picture, and the motion vector value can be set to (0,0), which is zero MV.

[0237] As another example, for a surrounding sub-block at position 12 where no motion information exists, a collocated picture in the L0 direction (or L1 direction) is assigned as a reference picture, and the motion vector value can be set to (0,0).

[0238] However, the present invention is not limited to the above examples, and pre-set default motion information may be set for surrounding sub-blocks that do not have motion information.

[0239] In another embodiment, if the surrounding sub-block is not encoded in an inter-prediction mode but in an Intra Block Copy (IBC) mode, the Block Vector (BV) value of the IBC mode can be used as motion information for the corresponding surrounding sub-block.

[0240] In addition, if a surrounding sub-block is encoded in IntraTMP (Intra Template Matching Prediction) mode, the BV (Block Vector) value of the IntraTMP mode can be used as motion information for that surrounding sub-block.

[0241]

[0242] In the step of deriving motion information of sub-blocks within the current block, motion information of one or more sub-blocks within the current block, which is the block to be decoded, can be obtained by using motion information of surrounding sub-blocks.

[0243] Here, the sbSMVP type may be determined according to the direction in which movement information is induced from surrounding sub-blocks, and for example, there may be five sbSMVP types corresponding to each of the five directions as shown in FIG. 9, but the present invention is not limited thereto.

[0244] Meanwhile, if the status information of surrounding sub-blocks required for a specific direction sbSMVP type is confirmed and all of the surrounding sub-blocks are available, the sbSMVP type can be set as available.

[0245] For example, in the case of the horizontal sbSMVP type shown in FIG. 9(a), the status of the surrounding sub-blocks located at numbers 11 to 14 is checked, and if all four surrounding sub-blocks are available, the horizontal sbSMVP type can be set as available.

[0246] And when the “horizontal” type is selected as the sbSMVP type of the sub-block-based merge mode in the encoding device, the decoder can copy the movement information of surrounding sub-blocks at positions 11 to 14 that are spatially adjacent to the current block and use it as the movement information of sub-blocks within the current block based on the horizontal axis.

[0247] Meanwhile, if the status information of surrounding sub-blocks required for a specific direction sbSMVP type is checked and some of the surrounding sub-blocks are in an unavailable state, the sbSMVP type may be set to be in an unavailable state.

[0248] For example, in the case of the vertical sbSMVP type shown in Fig. 9(b), the status of the surrounding sub-blocks located at positions 1 through 4 is checked, and if one or more of the four surrounding sub-blocks are in an unavailable state, the vertical sbSMVP type can be set as unavailable.

[0249] In another embodiment, the state information of surrounding sub-blocks required for a specific direction sbSMVP type is checked, and if some of the surrounding sub-blocks are in an unavailable state, a step of inducing movement information for the surrounding sub-blocks in an unavailable state may be additionally performed.

[0250] For example, for a surrounding sub-block in an unavailable state, movement information of at least one of other surrounding sub-blocks located above, below, to the left, and to the right of the surrounding sub-block may be set, or preset default movement information may be set.

[0251] In another embodiment, if a certain number or more of the surrounding sub-blocks required for a specific direction sbSMVP type are in an available state (or, a certain number or fewer of the surrounding sub-blocks are in an unavailable state), the sbSMVP type may be set to be in an available state.

[0252] Meanwhile, if fewer than a certain number of surrounding sub-blocks are available among the surrounding sub-blocks required for a specific direction sbSMVP type (or if more than a certain number of surrounding sub-blocks are unavailable), the sbSMVP type may be set to be unavailable.

[0253] And since the method of inducing movement information of sub-blocks within the current block using movement information of surrounding sub-blocks may be as described with reference to FIG. 9, a detailed explanation thereof will be omitted.

[0254]

[0255] In the step of adding sbSMVP candidates to the candidate list of sub-block-based merge modes, a candidate list for sub-block-based merge modes can be configured.

[0256] Here, the candidate list for the sub-block based merge mode may be entered in the order of sbTMVP candidates, sbSMVP candidates, and affine candidates, but the present invention is not limited thereto.

[0257] According to one embodiment of the present invention, the availability of each of the sbSMVP types according to a plurality of directions is checked for the current block, and one or more available sbSMVP types may be added to the candidate list as sbSMVP candidates. And for unavailable sbSMVP types, sbSMVP candidates of the corresponding type may not be added to the candidate list.

[0258] For example, as described above, availability can be checked for five sbSMVP types according to five directions, and the order of sbSMVP types added to the candidate list after availability is checked may be horizontal, vertical, lower right diagonal, upper right diagonal, and lower left diagonal as shown in FIG. 9, but the present invention is not limited thereto.

[0259] Meanwhile, when an sbSMVP candidate is added to the candidate list, an sbSMVP type that has duplicate movement information identical to movement information already added to the candidate list may not be added to the candidate list.

[0260] Here, duplicate movement information may mean that the movement information of each sub-block is identical for all sub-blocks within the current block.

[0261] For example, in the case illustrated in FIG. 9, if the surrounding sub-blocks at positions 11 to 14 and the surrounding sub-blocks at positions 1 to 4 all have the same movement information, the horizontal type sbSMVP candidate and the vertical type sbSMVP candidate are identical, so the horizontal sbSMVP candidate is added to the candidate list according to the order of adding to the candidate list, and the vertical sbSMVP candidate may not be added to the candidate list.

[0262] In another embodiment, when adding sbSMVP candidates to the candidate list, all available types of sbSMVP candidates may be added to the candidate list without performing a duplicate check for motion information as described above, in order to eliminate complexity associated with the duplicate check.

[0263] In another embodiment, sbSMVP candidates according to multiple directions are added to a candidate list in a predetermined order, and then an evaluation is performed on each sbSMVP candidate added to the candidate list so that the order of sbSMVP candidates in the candidate list can be reordered.

[0264] Here, the evaluation of sbSMVP candidates within the candidate list can be performed using the Template Matching (TM) method.

[0265] Specifically, the order (or position) of sbSMVP candidates within the candidate list can be adjusted so that lower TM costs have a higher order (e.g., lower index) and higher TM costs have a lower order (e.g., higher index).

[0266] For example, by using movement information of surrounding 4x4 size sub-blocks according to a specific direction, movement information for sub-blocks within the current block is obtained, and a reference block can be created according to the movement information.

[0267] At this time, for sub-blocks located on the left and upper sides within the current block, the values ​​of the reconstructed pixels for a certain area outside the reference block are additionally obtained to generate the template area (ref_nb_sub-blk_template) of the reference block.

[0268] For example, reconstructed pixels outside the adjacent reference block can be obtained by using movement information of sub-blocks located to the left and top of the current block.

[0269] And by calculating the error value between the template area already restored around the current block (curr_blk_template) and the template area of ​​the reference block according to a specific direction (ref_nb_sub-blk_template), the TM cost for the sbSMVP candidate of that direction can be obtained.

[0270] Here, since the movement information of each sub-block located on the left and upper side within the current block may differ according to a specific direction, the template area (ref_nb_sub-blk_template) of the reference block pointed to by the movement information of the sub-block may differ. In this case, error values ​​are calculated for each of the template areas for the sub-blocks, and these are summed (or averaged) to calculate the TM cost for the sbSMVP candidate of the direction.

[0271] And the order (or position) of sbSMVP candidates can be adjusted so that sbSMVP candidates with low TM costs within the candidate list have a higher order (e.g., lower index), and sbSMVP candidates with high TM costs have a lower order (e.g., higher index).

[0272] In this specification, the cost function for TM may be calculated using at least one of SAD (Sum of Absolute Differences), SATD (Sum of Absolute Transformed Differences), and MSE (Mean Squared Error), but the present invention is not limited thereto.

[0273] As another example, a refinement operation using TM can be performed on the movement information of the sub-block within the current block derived as described above.

[0274] Specifically, the motion vector of a sub-block within the current block can be corrected to a motion vector corresponding to the template area with the lowest TM cost within a certain area of ​​the reference picture for the sub-block.

[0275] For example, error values ​​are calculated between a template region (sub-block_template) that has already been restored around a sub-block within the current block and template regions (ref_sub-block_templates) within a certain area (R) of the reference picture for that sub-block, so that the template region with the lowest TM cost (i.e., the smallest error value) within the certain area (R) of the reference picture can be selected.

[0276] And, with the position indicated by the motion vector of the corresponding sub-block as the initial position, an offset value representing the displacement from that initial position to the position of the template area with the lowest TM cost can be determined.

[0277] Accordingly, the offset value determined as above is added to the motion vector of the corresponding sub-block, and a corrected motion vector can be obtained.

[0278] Meanwhile, the offset value determined as above is shared among all sub-blocks within the current block and can be added to the motion vector of each sub-block within the current block.

[0279] At this time, since there is already a restored template area (sub-block_template) around the sub-blocks located to the left and top of the current block, but no restored area around the sub-blocks located inside the current block, the offset value can be determined using the sub-blocks located to the left and top of the current block.

[0280] In another embodiment, priority can be set for sbSMVP candidates of a specific direction so that sbSMVP candidates of a specific direction can be excluded from evaluation when reordering sbSMVP candidates.

[0281] For example, by giving priority to horizontal sbSMVP candidates and vertical sbSMVP candidates, when reordering sbSMVP candidates, the order (or position) of horizontal and vertical sbSMVP candidates within the candidate list is not adjusted, and only the order (or position) of sbSMVP candidates in the remaining directions is adjusted through evaluation.

[0282] As another example, for a sub-block-based merge mode, the candidate list is entered in the order of sbTMVP candidates, sbSMVP candidates, and affine candidates, and then an evaluation is performed on all candidates in the candidate list to reorder the order of all candidates in the candidate list.

[0283] Here, the evaluation of all candidates in the candidate list can be performed using the TM method, and the evaluation may include at least some of the methods for reordering the order of sbSMVP candidates in the candidate list described above, and the description of the same as the method for reordering the order of sbSMVP candidates in the candidate list described above will be omitted below.

[0284]

[0285] In the sub-block based merge mode described above, motion compensation (MC) is performed by inducing motion information in units of small sub-blocks (e.g., 4x4), so the mode may be meaningful for decoding target blocks of a certain size or larger.

[0286] Accordingly, when the size of the block to be decrypted is greater than a certain size, the 'sub_block_merge_flag' syntax indicating whether to apply the sub-block-based merge mode may be transmitted for the block.

[0287] Meanwhile, for decryption target blocks smaller than a certain size, the sub-block based merge mode may be disabled.

[0288] For example, when the height and width of the block to be decoded are each 8 or greater, and at least one of the height and width values ​​exceeds 8, the sub-block-based merge mode function is enabled, and the 'sub_block_merge_flag' syntax can be transmitted for the block.

[0289] As another example, when the height and width of the block to be decoded are both 8 or greater, the sub-block-based merge mode function is enabled, and the 'sub_block_merge_flag' syntax can be transmitted for the block.

[0290] As another example, if the size (height × width) of the block to be decrypted is 32 or greater, the sub-block-based merge mode function is enabled, and the 'sub_block_merge_flag' syntax can be transmitted for the block.

[0291] As another example, when the height and width of the block to be decoded are each 8 or greater, and at least one of the height and width values ​​exceeds 8, the sub-block-based merge mode function is always enabled, and the 'sub_block_merge_flag' syntax can be transmitted for the block.

[0292] Meanwhile, if the above conditions are not satisfied and the size (height × width) of the block to be decoded is 32 or greater, it is checked whether spatially adjacent neighboring blocks to the block are encoded in sub-block-based merge mode, and if at least one of the neighboring blocks is encoded in sub-block-based merge mode, the function of sub-block-based merge mode is activated and the 'sub_block_merge_flag' syntax can be transmitted for the block.

[0293] And if the 'sub_block_merge_flag' value is "on" (or "1"), the 'merge_candidate_idx' value representing the selected candidate from the candidate list may additionally be signaled.

[0294]

[0295] Meanwhile, in a sub-block based merge mode, the block to be decoded is divided into 4x4 sub-blocks, and motion compensation can be performed using different motion information for each sub-block.

[0296] Specifically, a prediction block (or reference block) for a block to be decoded can be generated by using reconstructed pixels at different locations for each of the sub-blocks.

[0297] Accordingly, blocking artifacts, which are distortion phenomena, may occur at the boundaries corresponding to sub-blocks within the reference block, and to improve this, a blending filter or a smoothing filter may be used at the sub-block boundaries within the reference block.

[0298] In another embodiment, the Overlapped Block Motion Compensation (OBMC) process may be performed on the sub-block boundaries within the reference block.

[0299] Here, for each sub-block, the values ​​of the reconstructed pixels for the overlapped block are additionally calculated, and a motion compensation process can be performed using these.

[0300] For example, at the boundary of sub-blocks within a reference block, a reference block for motion compensation can be generated by performing a blending process on the reconstructed pixels of the boundary using motion information of one or more spatially adjacent neighboring sub-blocks.

[0301] In another embodiment, the artifact problem caused by sub-block boundaries within the reference block as described above may be improved by using a conventional DBF (De-Blocking Filter).

[0302] In another embodiment, different filtering methods may be applied depending on the directionality of sbSMVP as described above.

[0303] For example, blending filters, smoothing filters, OBMC, or DBF methods can all be applied to sbSMVPs of vertical and horizontal directions, but only blending filters, smoothing filters, or OBMC methods may be applied to sbSMVPs of other diagonal directions.

[0304]

[0305] According to one embodiment of the present invention, a Sub-Block Transform (SBT) may be applied as a transform method for a block to be decoded.

[0306] Generally, the transformation process can be performed by setting one transform type for each of the horizontal and vertical axes for the block to be decoded.

[0307] In contrast, in the case of SBT, the block to be decoded is divided into multiple sub-blocks, so that the conversion process is omitted for some of the multiple sub-blocks, and the conversion process is performed for some of the sub-blocks by setting one conversion type based on the horizontal axis and the vertical axis, respectively.

[0308] Here, some sub-blocks where the transformation process is omitted are cases where signal values ​​are absent or small enough that, after performing the prediction, the residual signal does not require transformation.

[0309] In other words, depending on the prediction results, some sub-blocks within the block to be decoded may have little or no residual signal, while others may have residual signal and require conversion.

[0310] Therefore, as a method is applied in which the block to be decoded is divided into multiple sub-blocks and prediction is performed on a sub-block basis, SBT may be a more advantageous transformation method for a specific decoded block.

[0311] Here, as a prediction method at the sub-block level, there may be intra prediction methods such as ISP (Intra Sub-Partition), and inter prediction methods such as affine, GPM, and sbTMVP, but the present invention is not limited thereto.

[0312] Table 3 below shows an example of the syntax of a coding unit related to SBT.

[0313]

[0314] Referring to Table 3, if the 'cu_sbt_flag' value of the block to be decoded is "on" (or "1"), SBT is performed on that block, and for this purpose, partition direction information and transform position information may be additionally transmitted.

[0315] Here, the splitting direction is specified by the syntax 'cu_sbt_horizontal_flag', and if the flag value is "on" (or "1"), the block is split into sub-blocks along the horizontal axis, and if the flag value is "off" (or "0"), the block is split into sub-blocks along the vertical axis.

[0316] The sub-block-based spatial motion vector prediction (sbSMVP) method according to the embodiment of the present invention as described above is a method that divides a block to be decoded into a plurality of sub-blocks and performs prediction on a sub-block basis, and the SBT as described above may be advantageous as a conversion method in the sbSMVP mode.

[0317] In addition, in sbSMVP mode, since there is a directionality to induce movement information from surrounding sub-blocks, information regarding that directionality can be utilized in SBT.

[0318] For example, in the case of a vertical sbSMVP, the 'cu_sbt_horizontal_flag' indicating the splitting direction for SBT is not signaled, and the flag value is set to "off" (or "0") so that the block is split into sub-blocks along the vertical axis for SBT.

[0319] Meanwhile, in the case of horizontal direction sbSMVP, the 'cu_sbt_horizontal_flag' indicating the division direction for SBT is not signaled, and the flag value is set to "on" (or "1") so that the block is divided into sub-blocks based on the horizontal axis for SBT.

[0320] As another example, for vertical sbSMVP, 'cu_sbt_horizontal_flag' is not signaled and is set to "on" (or "1"), and for horizontal sbSMVP, 'cu_sbt_horizontal_flag' is not signaled and may be set to "off" (or "0").

[0321] In another embodiment, when the block to be decoded is predicted to be in sbSMVP mode, 'cu_sbt_flag', a flag indicating whether SBT is used, is not signaled and is set to "on" (or "1"), and 'cu_sbt_horizontal_flag' is also not signaled and may be set according to the directionality of sbSMVP.

[0322]

[0323] The inter prediction method described above can be performed by an image encoding device and an image decoding device according to an embodiment of the present invention, and, for example, by an inter prediction unit (165) of a video encoder (100) shown in FIG. 1 or an inter prediction unit (265) of a video decoder (200) shown in FIG. 3.

[0324]

[0325] The methods described above in this specification may be performed through a processor of a video encoder or a video decoder. Additionally, the encoder may generate a bitstream that is decoded by a video signal processing method, and the bitstream generated by the encoder may be stored in a computer-readable non-transient storage medium (recording medium).

[0326] The embodiments of the present invention described above may be implemented through various means. For example, the embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

[0327] Some embodiments may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media.

[0328] Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules, or other transmission mechanisms, and include any information transmission media.

[0329] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. In a method for performing sub-block-based inter-prediction, A step of obtaining movement information of surrounding sub-blocks spatially adjacent to the current block; A step of inducing movement information of sub-blocks within the current block using the movement information of surrounding sub-blocks obtained above; and The method includes the step of adding a sub-block-based spatial motion vector prediction (sbSMVP) candidate to a candidate list for a sub-block-based merge mode using motion information of the above-described sub-blocks; An inter prediction method characterized by adding sbSMVP candidates corresponding to one or more available directions among a plurality of directions from which movement information is derived from the surrounding sub-blocks to the candidate list.

2. In Paragraph 1, An inter prediction method in which, for a surrounding sub-block not encoded in an inter prediction mode among the surrounding sub-blocks, movement information of another sub-block adjacent to the surrounding sub-block or preset default movement information is used.

3. In Paragraph 1, An inter prediction method in which, if at least one of the surrounding sub-blocks required for inducing movement information according to the above direction does not have movement information, the sbSMVP type of the direction is set to not-available.

4. In Paragraph 1, An inter prediction method in which, if motion information does not exist for a specific number or fewer of the surrounding sub-blocks required to induce motion information according to the above direction, the sbSMVP type of the corresponding direction is set as available.

5. In Paragraph 1, An inter-prediction method that adds candidates to the candidate list in the order of sub-block-based spatial motion vector prediction (sbSMVP) candidates, said sbSMVP candidates, and said affine merge candidates.

6. In Paragraph 1, The method further includes the step of performing a redundancy check on sbSMVP candidates corresponding to each of the above available directions, An inter-prediction method in which sbSMVP candidates with identical movement information for the above sub-blocks are not added redundantly to the above candidate list.

7. In Paragraph 1, An inter-prediction method further comprising the step of performing a re-ordering operation using Template Matching (TM) for sbSMVP candidates corresponding to each of the above available directions.

8. In Paragraph 1, An inter-prediction method further comprising the step of performing a refinement operation using TM on the movement information of the above-described induced sub-blocks.

9. An image encoding device that performs the method of any one of paragraphs 1 through 8.

10. An image decoding device that performs the method of any one of paragraphs 1 through 8.

Images