Subblock-based inter prediction method, and image encoding and decoding apparatus for performing same

Abstract

The present invention relates to a subblock-based inter prediction method, and an image encoding and decoding apparatus for performing same, and the method comprises the steps of: determining a collocated picture for the current block; determining a collocated block within the collocated picture; obtaining motion information of the determined collocated block; obtaining motion information of each subblock within the collocated block; and scaling a motion vector.

Description

Sub-block based inter-prediction method, image encoding and decoding device performing the same

[0001] The present invention relates to a sub-block-based inter-prediction method that enables efficient representation of motion information.

[0002] Compression encoding of video signals is achieved by removing redundant information by considering spatial correlations, temporal correlations, probabilistic correlations, etc.

[0003] Among the prediction methods used in video encoding, intra prediction is a method that predicts the current block by referring to already reconstructed samples around the block to be encoded.

[0004] Recently, due to advancements in various media and data transmission methods, user demand for high-resolution, high-quality video is increasing.

[0005] As encoded data for high-resolution images contains a greater amount of information than encoded data for low- or medium-resolution images, the cost of transmitting or storing it increases.

[0006] To address this problem, research on encoding and decoding methods to effectively reduce the amount of encoded data for high-resolution images is continuing.

[0007] The present invention aims to provide a sub-block-based inter-prediction method capable of improving image compression performance by efficiently representing motion information, and an image encoding and decoding device for performing the same.

[0008] An inter prediction method according to an embodiment of the present invention is a method for performing inter prediction based on sub-blocks, comprising: a step of determining a collocated picture for a current block; a step of determining a collocated block within the collocated picture; a step of obtaining movement information of the determined collocated block; a step of obtaining movement information of each sub-block within the collocated block; and a step of scaling the movement vector.

[0009] The above-mentioned scaled motion vector can be included in the candidate list for the sub-block-based merge mode.

[0010] Meanwhile, regarding the current block, a plurality of blocks at the same location can be set, and in the case of the B slice, two pictures at the same location corresponding to both directions can be set.

[0011] An image encoding device according to one embodiment of the present invention performs an inter prediction method as described above.

[0012] The image decoding device according to one embodiment of the present invention performs the inter prediction method as described above.

[0013] Meanwhile, at least some of the steps of the above methods may be implemented in a computer-readable recording medium that records a program for execution on a computer, and may also be provided as the program itself.

[0014] According to one embodiment of the present invention, by deriving motion vectors in sub-block units, motion vectors can be expressed more precisely, thereby improving the compression performance of the image.

[0015] Figure 1 is a block diagram illustrating the configuration and operation of a video encoder for encoding images.

[0016] FIG. 2 is a diagram illustrating an example of a method for dividing blocks of an image.

[0017] Figure 3 is a block diagram illustrating the configuration and operation of a video decoder for decoding an image.

[0018] FIG. 4a is a diagram illustrating an example of surrounding blocks spatially adjacent to the current block.

[0019] Figure 4b is a diagram illustrating an example of a method for scaling motion vectors.

[0020] FIG. 5 is a diagram illustrating embodiments of surrounding blocks spatially adjacent to the current block.

[0021] FIGS. 6 and FIGS. 7 are drawings for illustrating embodiments of a method for scaling motion vectors.

[0022] FIGS. 8 to 11 are drawings for illustrating embodiments of a method for determining identical location blocks.

[0023] FIG. 12 is a diagram illustrating an example of a method for refining the motion vector of a sub-block.

[0024] FIG. 13 is a diagram illustrating an example of a method for scaling the motion vector of a sub-block.

[0025] Hereinafter, an image encoding and decoding method and apparatus according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0026] In the following description of the present invention, specific descriptions of related known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, their definitions should be based on the overall content of the present invention.

[0027] In addition, the preferred embodiments of the present invention described below will focus on explaining the functional configurations that must be additionally provided for the present invention, while omitting as much as possible the system functional configurations that are already provided in each system functional configuration or are ordinarily provided in the technical field to which the present invention belongs, in order to efficiently explain the technical components constituting the present invention.

[0028] If a person skilled in the art to which the present invention pertains can easily understand the function of a component that has been used in the past among the functional configurations that are omitted and not illustrated below, and can also clearly understand the relationship between the component that was omitted as above and the component added for the present invention.

[0029] In this specification, a device that encodes an image to generate a video signal bitstream is referred to as an encoding device, an encoding device, or an encoder, and a device that decodes a video signal bitstream to restore an image is referred to as a decoding device, a decoding device, or a decoder.

[0030] A pixel or pel refers to the smallest unit that constitutes an image, and the terms pixel and sample may be used interchangeably. Generally, a sample can represent a pixel or its value, and it may represent only the pixel or its value of the luminance component, or only the pixel or its value of the chroma component.

[0031] Furthermore, the term "unit" is used to refer to a basic unit of image processing or a specific location within a picture, representing an image region containing at least one of the luminance component and the chrominance component. Specifically, the term "unit" can be used as a concept encompassing the Coding Tree Unit (CTU), Coding Unit (CU), Prediction Unit (PU), and Transform Unit (TU). Additionally, the term "block" represents an image region containing a specific component among the luminance and chrominance components, and an MxN block can represent a set of samples or transform coefficients consisting of M columns and N rows. Here, terms such as unit, block, partition, signal, and region may be used interchangeably.

[0032] Meanwhile, the term "picture" refers to a field or a frame, and these terms may be used interchangeably. For example, if the image is interlaced, a single frame is separated into an odd (or top) field and an even (or bottom) field, and each field is configured as a single picture unit for encoding or decoding. And if the image is sequential, a single frame is configured as a picture for encoding or decoding.

[0033]

[0034] FIG. 1 is a block diagram illustrating an encoding device according to an embodiment of the present invention, and is intended to explain the configuration and operation of a video encoder for encoding images.

[0035] Referring to FIG. 1, the video encoder (100) may be configured to include a conversion unit (110), a quantization unit (120), an inverse quantization unit (130), an inverse conversion unit (140), a filtering unit (150), a prediction unit (160), a DPB (Decoded Picture Buffer, 170), and an entropy coding unit (180).

[0036] The conversion unit (110) converts the residual signal, which is the difference between the input video signal and the prediction signal generated by the prediction unit (160), to obtain a conversion coefficient value.

[0037] For example, the Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), or Wavelet Transform may be used.

[0038] The transformation kernel used for the transformation of a residual block may be a transformation kernel having separable characteristics of vertical transformation and horizontal transformation. In this case, the transformation of the residual block can be performed by separating it into a vertical transformation and a horizontal transformation. For example, an encoder may perform a vertical transformation by applying a transformation kernel in the vertical direction of the residual block. Additionally, an encoder may perform a horizontal transformation by applying a transformation kernel in the horizontal direction of the residual block.

[0039] Meanwhile, a transformation kernel can be used as a term referring to a set of parameters used for transforming a residual signal, such as a transformation matrix, transformation array, transformation function, or transformation; it can be any one of a plurality of available kernels, and transformation kernels based on different transformation types may be used for vertical transformation and horizontal transformation, respectively.

[0040] Transformation coefficients may be distributed such that higher coefficients are found towards the top-left corner of the block, and coefficients close to '0' are found towards the bottom-right corner. Additionally, as the current block size increases, there is a higher likelihood of '0' coefficients existing in the bottom-right region; to reduce the transformation complexity of large blocks, only an arbitrary top-left region may be retained, while the remaining regions may be reset to '0'.

[0041] In addition, an error signal may exist only in some areas of a coding block, in which case the conversion process may be performed only on some arbitrary areas. For example, in a block of size 2Nx2N, an error signal may exist only in the first 2NxN block, in which case the conversion process is performed only on the first 2NxN block, but the second 2NxN block may not be encoded or decoded without the conversion process being performed.

[0042] The encoder may perform additional transformations before the transformation coefficients are quantized. The transformation method described above is referred to as a primary transform, and the additional transformation may be referred to as a secondary transform.

[0043] The second conversion can be optional for each residual block, for example, the encoder can improve coding efficiency by performing a second conversion for areas where it is difficult to concentrate energy in the low-frequency region with only the first conversion.

[0044] Specifically, a second transformation may be additionally performed on blocks where residual values ​​appear significantly in directions other than the horizontal or vertical direction of the residual block, and unlike the first transformation, the second transformation may not be performed separately into a vertical transformation and a horizontal transformation. The second transformation described above may be referred to as a Low Frequency Non-Separable Transform (LFNST).

[0045] The quantization unit (120) quantizes the conversion coefficient value output from the conversion unit (110).

[0046] In order to increase coding efficiency, instead of coding the picture signal as is, a method is used to predict the picture using an area that has already been coded through a prediction unit (160), and to obtain a restored picture by adding the residual value between the original picture and the predicted picture to the predicted picture.

[0047] To prevent mismatches from occurring between the encoder and decoder, when performing prediction in the encoder, information available in the decoder must be used; to this end, the encoder may perform a process of restoring the currently encoded block.

[0048] The inverse quantization unit (130) inversely quantizes the conversion coefficient value, and the inverse conversion unit (140) restores the residual value using the inversely quantized conversion coefficient value.

[0049] The filtering unit (150) performs filtering operations using a deblocking filter, a Sample Adaptive Offset (SAO), an Adaptive Loop Filter (ALF), etc., to improve the quality of the restored picture and enhance encoding efficiency.

[0050] A deblocking filter is a filter for removing distortion within blocks generated at the boundaries between blocks in a restored picture, and the encoder can determine whether to apply the deblocking filter to a boundary based on the distribution of pixels included in a few columns or rows relative to an arbitrary boundary within a block.

[0051] When a deblocking filter is applied, the filtering unit (150) can apply a long filter, a strong filter, or a weak filter depending on the deblocking filtering strength, and can process horizontal filtering and vertical filtering in parallel.

[0052] Sample Adaptive Offset (SAO) can be used to correct the offset from the original image on a pixel-by-pixel basis for a residual block to which a deblocking filter has been applied. The filtering unit (150) may use a Band Offset method to correct the offset for a specific picture by dividing the pixels included in the image into a certain number of regions, determining the region to perform offset correction, and applying the offset to the region. Additionally, the filtering unit (150) may use an Edge Offset method to apply the offset by considering the edge information of each pixel.

[0053] An Adaptive Loop Filter (ALF) is a method that divides pixels contained in an image into specific groups, determines a single filter to apply to each group, and performs differential filtering for each group. Information regarding whether to apply an Adaptive Loop Filter can be signaled at the coding unit level, and the shape and filter coefficients of the ALF filter to be applied may vary depending on each block. Additionally, the same type of Adaptive Loop Filter may be applied regardless of the characteristics of the current block to be applied.

[0054] The filtered picture can be stored in DPB (170) to be used as a reference picture.

[0055] The prediction unit (160) includes an intra / IBC prediction unit (161) and an inter prediction unit (165). The intra / IBC prediction unit (161) performs intra prediction and IBC prediction within the current picture, and the inter prediction unit (165) performs inter prediction by predicting the current picture using a reference picture stored in the DPB (170).

[0056] The intra / IBC prediction unit (161) performs intra prediction from the restored regions within the current picture and transmits intra encoding information to the entropy coding unit (180). Here, the intra encoding information may include at least one of an intra prediction mode, an MPM (Most Probable Mode) flag, an MPM index, and information regarding a reference sample.

[0057] Additionally, the intra / IBC prediction unit (161) can perform IBC (Intra Block Copy) prediction from the restored samples in the current picture and transmit the IBC encoding information to the entropy coding unit (180).

[0058] In this case, the intra / IBC prediction unit (161) obtains a block vector value indicating a reference area used for prediction of the current area by referring to a specific area within the current picture, and can perform IBC prediction using the obtained block vector value.

[0059] The intra / IBC prediction unit (161) can transmit IBC encoding information, including at least one of the size information of the reference area and block vector information (index information for predicting the block vector of the current block within the motion candidate list and block vector difference information), to the entropy coding unit (160).

[0060] The inter prediction unit (165) refers to a specific area of ​​the restored reference picture to find the part most similar to the current area and obtains a Motion Vector (MV) value, which is the distance between the areas, and transmits motion information for the obtained reference area (reference direction indicator information (L0 prediction, L1 prediction, bidirectional prediction), reference picture index, motion vector information, etc.) to the entropy coding unit (180).

[0061] Additionally, the inter prediction unit (165) performs motion compensation using motion information to generate a prediction block for the current block, and transmits inter encoding information containing motion information for a reference area to the entropy coding unit (180).

[0062] Meanwhile, quantized transformation coefficients in the form of a two-dimensional array can be rearranged into a one-dimensional array for entropy coding.

[0063] The method for scanning quantized transform coefficients can be determined by the size of the transform block and the intra-prediction mode, and diagonal, vertical, or horizontal scans can be applied, and the scan information can be signaled in block units or derived from a decoder according to a defined rule.

[0064] The entropy coding unit (180) generates a bitstream by entropy coding information representing quantized conversion coefficients, intra-coding information, and inter-coding information, etc. For this purpose, a variable length coding (VLC) method and an arithmetic coding method may be used.

[0065] Variable Length Coding (VLC) converts input symbols into a sequence of codewords, the length of which can be variable. For example, frequently occurring symbols can be represented by short codewords, while infrequently occurring symbols can be represented by long codewords.

[0066] As a variable-length coding method, Context-based Adaptive Variable Length Coding (CAVLC) can be used.

[0067] Arithmetic coding utilizes the probability distribution of each data symbol to convert consecutive data symbols into a single prime number, thereby obtaining the optimal prime bit required to represent each symbol.

[0068] As an arithmetic coding method, the Context-based Adaptive Binary Arithmetic Code (CABAC) method can be used.

[0069] CABAC is a method of binary arithmetic encoding using multiple context models generated based on probabilities obtained through experiments. A context model can also be referred to as a context model. First, if a symbol is not in binary form, the encoder binarizes each symbol using exp-Golomb or similar methods. Binarized 0s or 1s can be described as bins.

[0070] The CABAC initialization process is divided into context initialization and arithmetic coding initialization. Context initialization is the process of initializing the occurrence probability of each symbol, which is determined by the type of symbol, quantization parameters (QP), and slice type (whether it is I, P, or B). A context model containing this initialization information can use probability-based values ​​obtained through experiments.

[0071] The context model provides the probability of occurrence of LPS (Least Probable Symbol) or MPS (Most Probable Symbol) for the symbol currently to be coded, and information (valMPS) on which empty value between 0 and 1 corresponds to the MPS.

[0072] One of several context models is selected through the context index (ctxIdx), and the context index can be derived from information about the block currently to be encoded or information about surrounding blocks.

[0073] Initialization for binary arithmetic coding is performed based on the probability model selected from the context model. Binary arithmetic coding proceeds by dividing into probability intervals based on the occurrence probabilities of 0 and 1, and then the probability interval corresponding to the bin to be processed becomes the entire probability interval for the next bin to be processed.

[0074] The location information within the probability interval where the last bin was processed is output. However, since the probability interval cannot be divided indefinitely, if it shrinks to within a certain size, a renormalization process is performed to widen the probability interval and output the corresponding location information. Additionally, after each bin is processed, a probability update process may be performed to newly set the probability for the next bin to be processed based on the information of the processed bin.

[0075] As mentioned above, the generated bitstream is encapsulated with a Network Abstraction Layer (NAL) unit as the basic unit.

[0076] NAL units are classified into VCL (Video Coding Layer) NAL units containing video data and non-VCL NAL units containing parameter information for decoding video data, and various types of VCL or non-VCL NAL units may exist.

[0077] A NAL unit consists of NAL header information and data, which is a Raw Byte Sequence Payload (RBSP), and the NAL header information includes summary information about the RBSP. The RBSP of a VCL NAL unit contains encoded integer coding tree units.

[0078] In order to decode a bitstream in a decoder, the bitstream must first be separated into NAL units, and then each separated NAL unit must be decoded. Meanwhile, the information required for decoding the bitstream can be transmitted in the Picture Parameter Set (PPS), Sequence Parameter Set (SPS), Video Parameter Set (VPS), etc.

[0079] Meanwhile, the configuration and operation of the encoder described with reference to FIG. 1 are according to an embodiment of the present invention, and some configurations may be omitted or added as needed.

[0080]

[0081] Additionally, a single picture can be divided and encoded into sub-pictures, slices, tiles, etc. A sub-picture may include one or more slices or tiles. If a single picture is divided and encoded into multiple slices or tiles, it can be displayed on the screen only after all slices or tiles within the picture have been fully decoded.

[0082] When a single picture is encoded into multiple sub-pictures, any single sub-picture may be decoded and displayed on the screen. A slice may include multiple tiles or sub-pictures, and a tile may include multiple sub-pictures or slices.

[0083] Sub-pictures, slices, and tiles can be encoded or decoded independently of each other, which is effective for parallel processing and improving processing speed; however, since encoded information from adjacent sub-pictures, slices, and tiles cannot be utilized, the bit size may increase.

[0084] And sub-pictures, slices, and tiles can be divided into multiple Coding Tree Units (CTUs) and encoded.

[0085] The coding tree unit may consist of a 128x128 luminance coding tree block (Coding Tree Block, CTB) and two 64x64 color difference coding tree blocks.

[0086] A single coding tree unit may not be divided and may constitute a single coding unit (CU) itself, or it may be divided into multiple coding units as shown in FIG. 2. A coding unit may consist of a luminance coding block (CB) and two color difference coding blocks.

[0087] A single coding unit may consist of a single transform unit (TU) or be divided into multiple transform units. A transform unit may consist of a luminance transform block (TB) and two chrominance transform blocks.

[0088] Here, a coding unit represents a basic unit for processing a picture during the prediction, transformation, quantization, entropy coding, and decoding processes, and the size and shape of the coding unit within a single picture may not be constant.

[0089] The coding unit may have a square or non-square shape, and the non-square coding unit may include a vertical coding unit in which the height is greater than the width and a horizontal coding unit in which the width is greater than the height.

[0090] The coding tree unit is first partitioned into a Quad Tree (QT) structure, so that a single node of size 2NX2N can be partitioned into four nodes of size NXN. Meanwhile, the Quad Tree partitioning can be performed recursively, and not all nodes need to be partitioned to the same depth.

[0091] The leaf nodes of a quad tree can be further divided into a Multi-Type Tree (MTT) structure. For example, in a Multi-Type Tree structure, a single node can be divided into a binary or ternary tree structure of horizontal or vertical splitting. Accordingly, there can be four types of splitting structures in a Multi-Type Tree structure: vertical binary splitting, horizontal binary splitting, vertical ternary splitting, and horizontal ternary splitting.

[0092] In each tree structure, the width and height of the nodes can both have powers of 2. For example, in a binary tree (BT) structure, a node of size 2NX2N can be divided into 2 NX2N nodes by a vertical binary split and into 2 2NXN nodes by a horizontal binary split.

[0093] In addition, in a ternary tree (TT) structure, a node of size 2NX2N can be divided into (N / 2)X2N, NX2N, and (N / 2)X2N nodes by vertical ternary partitioning, and into 2NX(N / 2), 2NXN, and 2NX(N / 2) nodes by horizontal ternary partitioning. This multi-type tree partitioning can be performed recursively.

[0094] Leaf nodes of a multi-type tree can serve as coding units, and if a coding unit is not larger than the maximum transformation length, it can be used as a unit of prediction and transformation without further splitting. On the other hand, if the width or height of a coding unit is greater than the maximum transformation length, it can be split into multiple transformation units without explicit signaling regarding splitting.

[0095] The tree partitioning structure described above may have the same form (Single Tree) for the luminance block and the chrominance block, or different forms (Dual Tree) for the luminance block and the chrominance block.

[0096] Meanwhile, regarding the block division from the coding tree unit (CTU) to the coding unit (CU) as described above, the final coding unit can be determined by selecting the division structure with the smallest rate-distortion cost (RD cost) value within the allowable size and depth conditions through the rate-distortion optimization (RDO) process.

[0097] In addition, the syntax for expressing block partitions as described above may be as shown in Table 1 below.

[0098]

[0099] In Table 1, split_cu_flag indicates whether block partitioning has occurred or partitioning has ended, split_qt_flag indicates whether QT partitioning has occurred, mtt_split_cu_vertical_flag indicates whether MTT partitioning is vertical or horizontal, and mtt_split_cu_binary_flag indicates whether MTT partitioning is binary or ternary.

[0100]

[0101] FIG. 3 is a block diagram illustrating a decoding device according to an embodiment of the present invention, and is intended to explain the configuration and operation of a video decoder for encoding images.

[0102] Referring to FIG. 3, the video decoder (200) may be configured to include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transform unit (230), a filtering unit (240), a prediction unit (260), and a DPB (Decoded Picture Buffer, 270).

[0103] The entropy decoding unit (210) entropies decodes the bitstream to extract conversion coefficient information, intra-coding information, inter-coding information, etc. for each region.

[0104] For example, the entropy decoding unit (210) can obtain a binary code for conversion coefficient information of a specific region from the bitstream, and can obtain quantized conversion coefficients by debinding the binary code.

[0105] The inverse quantization unit (220) inversely quantizes the quantized conversion coefficients, and the inverse conversion unit (230) restores the residual value using the inversely quantized conversion coefficients.

[0106] Meanwhile, the residual value obtained from the inverse conversion unit (230) is added to the predicted value obtained from the prediction unit (260) to restore the original pixel value.

[0107] The filtering unit (240) performs filtering operations using a deblocking filter, a sample adaptive offset, an adaptive loop filter, etc., to improve the quality of the restored picture, and the filtered picture can be stored in the DPB (270) to be output or used as a reference picture for the next picture.

[0108] The prediction unit (260) includes an intra / IBC prediction unit (261) and an inter prediction unit (265), and generates a prediction picture by utilizing the encoding type decoded through the entropy decoding unit (210), the conversion coefficient for each region, intra / inter encoding information, etc.

[0109] To restore the current block being decrypted, the decrypted regions of the current picture containing the current block or other pictures may be utilized. A picture (or tile / slice) that performs intra prediction or intra-BC prediction using only the current picture for restoration is called an intra picture or I picture (or tile / slice), and a picture (or tile / slice) capable of performing intra prediction, inter prediction, and intra-BC prediction is called an inter picture (or tile / slice).

[0110] Meanwhile, a picture (or tile / slice) that uses at most one motion vector and reference picture index to predict sample values ​​of each block among the inter-pictures (or tiles / slices) is called a predictive picture or P picture (or tile / slice), and a picture (or tile / slice) that uses at most two motion vectors and reference picture indices is called a bi-predictive picture or B picture (or tile / slice).

[0111] That is, P picture (or tile / slice) uses at most one set of motion information to predict each block, and B picture (or tile / slice) uses at most two sets of motion information to predict each block. Here, a set of motion information may include one or more motion vectors and one reference picture index.

[0112] The intra / IBC prediction unit (261) generates a prediction block using intra-coding information and restored samples within the current picture, and the intra-coding information may include at least one of an intra-prediction mode, an MPM (Most Probable Mode) flag, and an MPM index.

[0113] The intra / IBC prediction unit (261) can predict the sample values ​​of the current block by using the restored samples located to the left and / or above the current block as reference samples.

[0114] For example, the reference samples may be samples adjacent to the left boundary of the current block and / or samples adjacent to the upper boundary, and among the samples of the surrounding blocks of the current block, samples located on a line within a preset distance from the left boundary of the current block and / or samples located on a line within a preset distance from the upper boundary of the current block. In this case, the surrounding blocks of the current block may include at least one of the left (L) block, upper (A) block, lower-left (BL) block, upper-right (AR) block, or upper-left (AL) block adjacent to the current block.

[0115] Additionally, the intra / IBC prediction unit (261) can restore the current region by referring to a specific region containing restored samples within the current picture. To this end, the intra / IBC prediction unit (261) can perform IBC prediction using IBC encoding information obtained from the entropy decoding unit (210), and the IBC encoding information may include block vector information.

[0116] The inter prediction unit (265) generates a prediction block using reference picture and inter encoding information stored in the DPB (270), and the inter encoding information may include a set of motion information of the current block for the reference block (reference picture index, motion vector, etc.).

[0117] Meanwhile, inter-prediction can include L0 prediction, L1 prediction, and bi-prediction.

[0118] L0 prediction means a prediction using one reference picture included in the L0 picture list, and L1 prediction means a prediction using one reference picture included in the L1 picture list. To do this, a set of motion information (e.g., motion vectors and reference picture indices, etc.) may be required.

[0119] In the pair prediction method, up to two reference regions can be used, and the two reference regions may exist in the same reference picture or in different pictures. Accordingly, in the up to two sets of motion information used in the pair prediction method, two motion vectors may correspond to the same reference picture index or to different reference picture indices.

[0120] At this time, the reference pictures are pictures located temporally before or after the current picture, and can be completed pictures that have already been restored, and the two reference regions used in the pair prediction method can be regions selected from the L0 picture list and the L1 picture list, respectively.

[0121] The inter prediction unit (265) can obtain a reference block of the current block using a motion vector and a reference picture index, and the reference block exists within a reference picture corresponding to the reference picture index.

[0122] In addition, a sample value of a block specified by a motion vector or an interpolated value thereof can be used as a predictor for the current block. For motion prediction with subpel-level pixel accuracy, an 8-tap interpolation filter can be used for the luminance signal and a 4-tap interpolation filter can be used for the chrominance signal.

[0123] Meanwhile, the configuration and operation of the decoder described with reference to FIG. 3 is according to an embodiment of the present invention, and some configurations may be omitted or added as needed, and the decoder can decode an image by performing the reverse process of the encoding method of the encoder described above.

[0124]

[0125] Hereinafter, the inter prediction method according to the present invention will be described in detail.

[0126] Inter prediction modes may include Skip / Merge mode and AMVP (Advanced Motion Vector Prediction) mode.

[0127] In skip / merge mode, a candidate list is constructed using movement information of spatially or temporally adjacent neighboring blocks, and the movement information of the current block can be represented using the index value of the selected movement information within the candidate list.

[0128] Meanwhile, merge modes can be divided into regular merge and sub-block merge modes.

[0129] In general merge mode, a merge candidate list is configured, and movement compensation can be performed by using the movement information of a specific candidate pointed to by an index among the merge candidate list as the movement information of the corresponding block (CU, Coding Unit).

[0130] Specifically, the merge candidate list may be composed of Spatial Merge Candidates, Temporal Motion Vector Prediction (TMVP) candidates, History-based Motion Vector Prediction (HMVP) candidates, Pairwise Average Candidates, and Zero Motion Vector (Zero MV).

[0131] Here, Temporal Motion Vector Prediction (TMVP) is a technique that derives motion information from collocated blocks within a reference picture existing at a different temporal location from the current picture and adds it to a merge candidate list or AMVP candidate list.

[0132] In Temporal Motion Vector Prediction (TMVP), a collocated picture, which is one of the reference pictures, is determined, and a collocated block within the collocated picture can be determined.

[0133] For example, information about collocated pictures and information about the predicted direction can be determined on a slice basis and included in a slice header as shown in Table 2 below.

[0134]

[0135] Referring to Table 2, the prediction direction information is specified by the 'sh_collocated_from_l0_flag' syntax of the slice header, and the index of the same location picture can be specified by the 'sh_collocated_ref_idx' syntax.

[0136] For example, a picture at the same location can be determined as the reference picture with index "2" among the reference pictures in reference picture list 0 (L0) when 'sh_collocated_from_l0_flag' is "1" and 'sh_collocated_ref_idx' is "2", and as the reference picture with index "3" among the reference pictures in reference picture list 1 (L1) when 'sh_collocated_from_l0_flag' is "0" and 'sh_collocated_ref_idx' is "3".

[0137] Meanwhile, the collocated block can be determined as a surrounding block (BR) located at the bottom-right relative to the block at the same location as the current block within the collocated picture, as shown in FIG. 4a.

[0138] Here, if the surrounding block (BR) at the bottom right is not available for reasons such as being located outside the picture boundary, or if it is encoded using an intra-prediction method, the block (CT) corresponding to the center position relative to the block at the same position as the current block within the same position picture may be determined as the same position block.

[0139] And in the step of scaling the motion vector, the motion vector of the same position block can be converted into the motion vector corresponding to the current block.

[0140] Here, when a TMVP candidate is added to the merge candidate list, the reference picture of the current block is set to the same location picture, and when a TMVP candidate is added to the AMVP candidate list, the reference picture of the current block can be explicitly signaled.

[0141] Referring to (a) of Fig. 4b, after scaling based on the Picture Order Count (POC) difference (curr_Poc_diff), which is the distance between the current picture (curr_pic) and the same-location picture (col_pic), and the POC difference (col_Poc_diff), which is the distance between the same-location picture (col_pic) and the reference picture (col_ref_pic) of the same-location block (col_blk), it can be included in the merge candidate list as a TMVP candidate for the current block (curr_blk).

[0142] For example, the scaling factor applied to the motion vector of the same location block (col_blk) can be calculated as the ratio between two POC difference values ​​(curr_Poc_diff, col_Poc_diff) as shown in Equation 1 below.

[0143]

[0144] Meanwhile, referring to (b) of FIG. 4b, the current block (curr_blk) can be included in the AMVP candidate list as a temporal TMVP candidate after scaling based on the difference in Picture Order Count (POC) (curr_Poc_diff) between the current picture (curr_pic) and the reference picture (curr_ref_pic) of the current block (curr_blk), and the difference in POC between the same location picture (col_pic) and the same location block (col_blk) (col_ref_pic).

[0145] For example, the scaling factor applied to the motion vector of the same location block (col_blk) can be calculated as the ratio between two POC difference values ​​(curr_Poc_diff, col_Poc_diff) as shown in Equation 2 below.

[0146]

[0147] Meanwhile, in addition to the general merge mode described above, MMVD (Merge with Motion Vector Difference) mode, CIIP (Combined Inter and Intra Prediction) mode, and TRIANGULAR mode can be used as inter prediction modes.

[0148] In sub-block based merge mode, the current block can be divided into multiple sub-blocks of a certain size, motion vectors can be calculated on a sub-block basis, and motion compensation for the corresponding sub-block can be performed using the motion vectors of each sub-block.

[0149] Meanwhile, the sub-block-based merge candidate list may include sub-block Temporal Motion Vector Prediction (sbTMVP) candidates and affine merge candidates with an affine motion model applied.

[0150] If the candidate index value for the motion information of the current block refers to the sbTMVP candidate among the sub-block-based merge candidate lists, a temporal motion vector can be derived at the sub-block level and motion compensation can be performed.

[0151] In sub-block-based temporal motion vector prediction (sbTMVP), collocated blocks within a collocated picture are divided into multiple sub-blocks, and motion vectors for each sub-block of the collocated blocks can be derived.

[0152] After that, the motion vectors derived for each sub-block of the same location block are mapped to values ​​corresponding to the reference picture of the current block and can be used as motion vectors for each sub-block of the current block.

[0153] Specifically, the sub-block-based temporal motion vector prediction (sbTMVP) method may sequentially include the steps of determining the location of blocks at the same location, obtaining representative motion information of blocks at the same location, obtaining motion information for each sub-block, and scaling motion vectors.

[0154] First, in the step of determining the location of a block at the same location, the location indicated by the movement information of a surrounding block adjacent to the current block or the same location within the picture at the same location can be determined as the location of the block at the same location.

[0155] Here, information about the same location picture may be included in the slice header as described above with reference to Table 2.

[0156] For example, referring to FIG. 5, by using the movement information of the block (A1) located on the left among the surrounding blocks of the current block, the block at the position indicated by the corresponding movement vector within the same position picture can be determined as the same position block.

[0157] Specifically, to determine the same location block, the prediction mode of the surrounding block (A1) of the current block is checked, and it can be determined whether the surrounding block (A1) is encoded in an inter-prediction mode.

[0158] If the surrounding block (A1) is encoded in an inter-prediction mode and the reference picture of the reference picture list 0 (L0) for the surrounding block (A1) is the same as the same location picture of the current block (first condition), then the block at the location pointed to by the motion vector in the L0 direction for the surrounding block (A1) within the same location picture can be determined as the same location block.

[0159] Meanwhile, if the above first condition is not satisfied, the surrounding block (A1) is encoded in an inter-prediction mode, and if the reference picture of the reference picture list 1 (L1) for the surrounding block (A1) is the same as the picture at the same location of the current block (second condition), the block at the location pointed to by the movement vector in the L1 direction for the surrounding block (A1) within the picture at the same location can be determined as the same location block.

[0160] And if the above first and second conditions are not satisfied, a block at the same location as the current block within the same location picture may be determined as the same location block.

[0161] Next, in the step of obtaining representative motion information of a block at the same location, if a collocated sub-block corresponding to a central position within the block at the same location is encoded in an inter-prediction mode, the motion vector of the collocated sub-block at the same location can be set as the representative motion vector (base MV) of the block at the same location.

[0162] Specifically, the prediction mode of a sub-block at the same location corresponding to the central location within the block at the same location is identified, and in this case, the sub-block at the same location may be a block encoded in an inter-prediction mode of not only unidirectional but also bidirectional.

[0163] If the corresponding same-location sub-block is encoded in an inter-prediction mode and motion information for the corresponding same-location sub-block exists, the sbTMVP mode is available, and the motion vector of the corresponding same-location sub-block can be set as the representative motion vector (base MV) of the same-location block.

[0164] Meanwhile, if the same location sub-block is not encoded in the inter-prediction mode and there is no movement information for the same location sub-block, the sbTMVP mode may not be available.

[0165] Next, in the step of obtaining movement information for each sub-block, the movement vector of each sub-block within the same location block can be set to a movement vector corresponding to the center position of the corresponding sub-block.

[0166] Here, the motion vector corresponding to the central position of the sub-block may mean a motion vector within a motion information storage (e.g., MV memory) that covers the central sample position of the sub-block.

[0167] Specifically, the prediction mode of each sub-block within the same location block can be checked first.

[0168] If the corresponding sub-block is encoded in an inter-prediction mode and motion information for the corresponding sub-block exists, a motion vector in the MV memory covering the central sample position of the corresponding sub-block can be set as the motion vector of the corresponding sub-block.

[0169] Meanwhile, if the sub-block is not encoded in an inter-prediction mode and there is no motion information for the sub-block, the representative motion vector (base MV) of the same location block can be set as the motion vector of the sub-block.

[0170] And in the step of scaling motion vectors, the motion vectors of each sub-block within the same location block can be converted into motion vectors corresponding to the current block.

[0171] Referring to Fig. 6(a), the position of a block (col_blk) in the same position within a picture (col_pic) is determined by using movement information for a surrounding block (nb_blk) of a current block (curr_blk) within a current picture (curr_pic), and movement information of each sub-block within a block (col_blk) in the same position can be obtained.

[0172] After that, the motion vector of each sub-block within the same-location block (col_blk) can be scaled using the POC difference (curr_Poc_diff) between the current picture (curr_pic) and the reference picture (curr_ref_pic) of the current block (curr_blk), and the POC difference (col_Poc_diff) between the same-location picture (col_pic) and the reference picture (col_ref_pic) of the same-location block (col_blk).

[0173] For example, the scaling factor applied to the motion vector of each sub-block within the same location block (col_blk) can be calculated as the ratio between two POC difference values ​​(curr_Poc_diff, col_Poc_diff), as shown in Equation 2 above.

[0174] Referring to Fig. 6(b), since the sub-block-based temporal motion vector prediction (sbTMVP) is added to the sub-block-based merge candidate list, when the reference picture (curr_ref_pic) of the current block (curr_blk) is set to the same location picture (col_pic), the scaling factor applied to the motion vector of each sub-block within the same location block (col_blk) may be calculated as the ratio between the two POC difference values ​​(curr_Poc_diff, col_Poc_diff) as in Equation 1 above.

[0175] Meanwhile, in the sub-block-based temporal motion vector prediction (sbTMVP) method described above, the size of the sub-block may be the same as the size unit in which motion information of the blocks is stored in a motion information storage (e.g., MV memory or buffer) to be used for future temporal motion vector prediction (TMVP).

[0176] For example, if motion information is stored in MV memory in units of 8x8, the size of the sub-block for sbTMVP may be 8x8, and if motion information is stored in MV memory in units of 4x4, the size of the sub-block for sbTMVP may be 4x4.

[0177] Additionally, the reference picture (curr_ref_pic) of the current block (curr_blk) for sbTMVP may be the picture corresponding to the 0th index of the reference picture list indicated by the syntax 'sh_collocated_from_l0_flag' in the slice header.

[0178] Referring to FIG. 7, the surrounding block (A1) of the current block in the current picture is encoded in an inter-prediction mode, and when the reference picture of the surrounding block (A1) is the same as the collocated picture for the current picture, the block at the position pointed to by the motion vector of the surrounding block (A1) in the collocated picture is set as the collocated block, and motion vector shifting (MV shifting) from the position of the current block in the current picture to the collocated block in the collocated picture can occur based on the movement of the surrounding block (A1).

[0179] In addition, blocks at the same location are divided into multiple sub-blocks, and the central location of blocks at the same location may be a location indicated by a dot.

[0180] Meanwhile, motion information within the MV memory covering the central location sample of the same location block can be set as the representative motion vector (base MV) for the same location block.

[0181] And when setting motion vectors for each sub-block within the same location block, for sub-blocks where motion information covering the central location sample of each sub-block is not in the MV memory, a representative motion vector (base MV) can be set as the motion vector of that sub-block.

[0182] The motion vector of each sub-block within the same location block is mapped (MV mapping) to correspond to the reference picture of the current block to be decoded, and can be used as the motion vector for each sub-block within the current block.

[0183] AMVP mode is an inter-prediction mode that transmits the Motion Vector Difference (MVD), which is the difference between the motion vector of the current block and the Motion Vector Prediction Value (MVP) derived from surrounding blocks.

[0184] Here, the AMVP Candidate List may be composed of Spatial Merge Candidates, Temporal Motion Vector Prediction (TMVP) Candidates, History-based Motion Vector Prediction (HMVP) Candidates, and Zero Motion Vector (Zero MV).

[0185] Meanwhile, in addition to the regular AMVP mode described above, the symmetric MVD mode and the AFFINE AMVP mode can be used as inter-prediction modes.

[0186] In addition, decoder-based motion vector correction techniques such as DMVR (Decoder-side MV Refinement), BDOF (Bi-Directional Optical Flow), and PROF (Prediction Refinement with Optical Flow) can be additionally applied to merge mode and AMVP mode.

[0187]

[0188] A sub-block-based inter-prediction method according to an embodiment of the present invention may include a process of deriving motion vector candidates from spatially and temporally adjacent neighboring blocks for each of the sub-blocks of a block to be decoded.

[0189] Specifically, a sub-block-based inter-prediction method according to an embodiment of the present invention comprises: a step of determining a collocated picture for a current block; a step of determining a collocated block within the collocated picture; a step of obtaining movement information of the determined collocated block; a step of obtaining movement information of each sub-block within the collocated block; and a step of scaling a movement vector; wherein a plurality of the collocated blocks may be set for the current block.

[0190] Here, the scaled motion vector may be included in the candidate list for the sub-block-based merge mode.

[0191] For B slice, two pictures of the same location corresponding to both directions may be set.

[0192] And the above same location picture can be adaptively set according to the reference picture of a surrounding block spatially adjacent to the current block.

[0193] When surrounding blocks spatially adjacent to the current block are encoded in a bidirectional inter-prediction mode, two pictures of the same location corresponding to both directions can be set based on the movement information of the surrounding blocks.

[0194] Meanwhile, the method may further include a step of performing an evaluation using TM (Template Matching) on ​​the same location block determined above, and a step of performing a refinement using TM (Template Matching) on ​​the movement information of the obtained sub-block.

[0195] The above sub-block-based inter-prediction method can be performed by an image encoding device or an image decoding device according to an embodiment of the present invention.

[0196]

[0197] The sub-block-based inter prediction method described below may include at least some of the inter prediction methods described above with reference to FIGS. 1 to 7, and the description of the same as the inter prediction method described above will be omitted below.

[0198] According to one embodiment of the present invention, a sub-block-based merge candidate list may include motion vector candidates of the following four groups, and when configuring the candidate list, motion vector candidates of one or more of the following four groups may be used.

[0199] 1) Sub-block based spatial motion vector prediction (sbSMVP) candidates

[0200] 2) Sub-block based temporal motion vector prediction (sbTMVP) candidates

[0201] 3) Affine Merge Candidate

[0202] 4) GPM (Geometric Partitioning Mode) Merge Candidates

[0203] Hereinafter, embodiments of a method for deriving motion vector candidates from spatially and temporally adjacent neighboring blocks for each sub-block of the current block, which is the block to be decoded, are described.

[0204] However, the motion vector candidate derivation process at the sub-block level according to the embodiment of the present invention is not limited to the sub-block-based merge mode and can also be used when constructing a candidate list for the regular merge mode, MMVD (Merge with Motion Vector Difference), and / or AMVP mode.

[0205] Meanwhile, a group of sub-block based merge modes including a regular sub-block merge mode, an affine sub-block merge mode, and / or a GPM sub-block merge mode may be configured, and when configuring a candidate list for the regular sub-block merge mode among the group of sub-block based merge modes, sbSMVP candidates and / or sbTMVP candidates may be used.

[0206] Here, up to k sbTMVP candidates are derived and can be included in the candidate list for sub-block based merge mode.

[0207] For example, multiple sbTMVP candidates may be derived, and up to four non-overlapping motion vector candidates may be added to the candidate list.

[0208] As mentioned above, the collocated picture serves as the reference picture that serves as the basis for Temporal Motion Vector Prediction (TMVP) for the current block. Accordingly, the collocated picture uses a specific picture within the reference picture list of the slice to which the current block belongs, and the index value designating the collocated picture can be the same as the index value within the reference picture list.

[0209] In addition, if a picture is divided into multiple slices and all divided slices use the same reference picture list, the same picture at the same location can be used for TMVP in all slices.

[0210] Meanwhile, if a single picture is divided into multiple slices and not all divided slices use the same reference picture list, one entry from the reference picture list in each slice may be used for TMVP. In this case, not all slices use the same picture at the same location, and different pictures at the same location may be assigned to each slice.

[0211] As a standard for setting the reference picture list for a slice, there may be a picture-level where the reference picture list is set based on a picture, and a slice-level where the reference picture list is set based on a slice.

[0212] In the case of the above picture level, since all slices split from a single picture use the same reference picture list, the same picture at the same location can be used for TMVP in all slices.

[0213] Meanwhile, in the case of the above slice level, since slices divided from a single picture can use different reference picture lists, different same-location pictures can be used for TMVP in each slice.

[0214] In addition, for the above picture level, in order to determine the number of reference pictures in the reference picture list, the number of reference pictures actually used is determined by the number of entries in the reference picture list, and the index of the picture at the same location can be signaled or set according to the number of entries in the reference picture list. Here, the number of entries in the reference picture list (num_ref_entries) can be signaled through High-Level Syntax (HLS) such as Sequence Parameter Set (SPS), Picture Parameter Set (PPS), or Picture Header (PH), and is related to the size of the memory (Decoded Picture Buffer, DPB) that stores the decoded picture.

[0215] Meanwhile, in the case of the above slice level, the index of the same location picture may be signaled or set according to the number of active reference pictures actually used in the slice, rather than the number of entries in the reference picture list. Here, the number of active reference pictures actually used in each slice (sh_num_ref_idx_active_minus1) is specified in the slice header (SH, Slice Header) as an HLS and can be set within the range of 0 to the number of entries in the reference picture list.

[0216] As mentioned above, the number of reference pictures in the reference picture list, which is the criterion for signaling or setting the index of a picture at the same location, refers to the entry in the reference picture list based on the picture, and may refer to the number of active reference pictures based on the slice.

[0217]

[0218] A method for deriving candidates for sub-block-based temporal motion vector prediction (sbTMVP) according to an embodiment of the present invention may include the steps of determining a collocated picture, determining a collocated block, obtaining motion information of a collocated block, obtaining motion information of a sub-block, and scaling the motion vector (MV scaling).

[0219] Below, examples for each step of the sbTMVP candidate induction method will be described in detail.

[0220]

[0221] According to one embodiment of the present invention, two identical position pictures corresponding to both directions may be used. Here, the two directions are not limited to backward and forward directions with respect to the time axis of the current picture, and may be two backward directions or two forward directions.

[0222] Meanwhile, for the P slice, one picture at the same location is used, and for the B slice type, two pictures at the same location may be used.

[0223] That is, the number of pictures in the same location can be determined according to the number of reference picture lists for the slice. For example, all blocks to be decoded within a slice that use only one of reference picture list 0 (L0) and reference picture list 1 (L1) use one picture in the same location, and all blocks to be decoded within a slice that use reference picture list 0 (L0) and reference picture list 1 (L1) can use two pictures in the same location.

[0224] Here, the number of reference picture lists used in the slice can be determined based on the number of entries in the corresponding reference picture list.

[0225] For example, if the number of entries in reference picture list 1 (L1) is "0", it may be determined that the reference picture list is not being used, and the slice may use one picture at the same location in reference picture list 0 (L0).

[0226] As another example, the number of reference picture lists used in a slice can be determined based on the number of active reference pictures actually used in that slice.

[0227] For example, if the number of active reference pictures in reference picture list 1 (L1) is "0", it may be determined that the reference picture list is not being used, and the slice may use one picture at the same location in reference picture list 0 (L0).

[0228] The step of determining the same-location pictures described below is not limited to the number of same-location pictures.

[0229] In the step of determining the same location picture, one or two same location pictures may be signaled or induced.

[0230] Here, when two collocated pictures are used, the two collocated pictures can be represented as the first collocated picture and the second collocated picture, respectively.

[0231] For example, one reference picture in reference picture list 0 (L0) may be set as the first picture at the same location, and one reference picture in reference picture list 1 (L1) may be set as the second picture at the same location, but the present invention is not limited thereto, and reference pictures at various locations may be used as pictures at the same location.

[0232] Meanwhile, information regarding a picture at the same location may be set at the picture or slice level and included in the header (PH, Picture Header) for the corresponding picture and / or the header (SH, Slice Header) for the corresponding slice as High-Level Syntax (HLS).

[0233] That is, a picture in the same location is set in the reference picture list for the picture (or slice) and signaled through the HSL, for example, when one picture in the same location is used, one reference picture in reference picture list 0 (L0) or reference picture list 1 (L1) is set as the picture in the same location, and when two pictures in the same location are used, one reference picture in reference picture list 0 (L0) and one reference picture in reference picture list 1 (L1) can be set as the picture in the same location.

[0234] Here, the reference picture (sh_collocated_ref_idx_from_L0) in reference picture list 0 (L0) may be the first picture at the same location, and the reference picture (sh_collocated_ref_idx_from_L1) in reference picture list 1 (L1) may be the second picture at the same location.

[0235] Meanwhile, when the TMVP function is enabled (hls_temporal_mvp_enabled_flag = "1"), information about the same location picture can be signaled as shown in Table 3 below.

[0236]

[0237] Referring to Table 3, the 'hls_temporal_mvp_enabled_flag' syntax can be included in one or more of the SPS (sequence parameter set), PPS (picture parameter set), picture header (PH), and slice header (SH) as an HLS.

[0238] If the number of reference pictures in reference picture list 0 (L0) and / or reference picture list 1 (L1) is one, the reference picture can be set to the same location picture without separate signaling.

[0239] That is, if the number of entries in reference picture list 0 (L0) and / or reference picture list 1 (L1) is "1", the reference picture can be set to the same location picture without separate signaling.

[0240] As another example, if the number of active reference pictures in reference picture list 0 (L0) and / or reference picture list 1 (L1) is "1", the reference picture can be set to the same location picture without separate signaling.

[0241] Meanwhile, if the number of reference pictures in reference picture list 0 (L0) and / or reference picture list 1 (L1) is 2 or more (NumRefIdxActive[0 or 1] > 1), picture information at the same location may be signaled.

[0242] In another embodiment, the same-location picture information is not signaled as header information (e.g., PH and / or SH), and the reference picture closest in time to the current picture within the current picture's reference picture list (L0 and / or L1) may be set as the same-location picture. In this case, the reference picture located at the closest POC distance to the current picture may be used as the same-location picture.

[0243] In another embodiment, the same location picture information is not signaled as header information (e.g., PH and / or SH), and the reference picture corresponding to the "0"th index value within the current picture's reference picture list (L0 and / or L1) can be set as the same location picture.

[0244] Here, a reference picture in reference picture list 0 (L0) can be set as the first picture in the same location, and a reference picture in reference picture list 1 (L1) can be set as the second picture in the same location.

[0245] As another example, whether picture information at the same location is induced or signaled may be determined by a flag for selectively controlling this. For example, if the value of 'sh_collocated_ref_idx_present_flag' is “0”, picture information at the same location may be induced. Conversely, if the value of 'sh_collocated_ref_idx_present_flag' is “1”, information regarding the picture at the same location may be explicitly signaled. Here, the method of inducing or signaling picture information at the same location refers to other examples of the present invention.

[0246] Although the above description explains an embodiment of the present invention using an example where a picture at the same position is fixed based on the current picture (or slice), the picture at the same position may be changed variably.

[0247] In this case, the same location picture is not fixed at the picture or slice level, but can be variably set at the block level to be decoded.

[0248] For example, a reference picture for a neighboring block of the current block can be adaptively set to a picture of the same location for the current block. In this case, for slice B, the number of pictures of the same location for the current block can be determined according to the prediction mode of the neighboring block.

[0249] Specifically, if the prediction mode of the surrounding block is a unidirectional inter-prediction mode, one identical location picture is set for the current block, and if the prediction mode of the surrounding block is a bidirectional inter-prediction mode, two identical location pictures can be set for the current block.

[0250] Here, the surrounding block is set as a single block at a specific location adjacent to the current block, and can be determined from among the surrounding blocks adjacent to the current block based on a preset location and / or order.

[0251]

[0252] According to one embodiment of the present invention, a plurality of sbTMVP candidates may be used, and accordingly, a plurality of blocks at the same location may also be set. That is, a plurality of blocks at the same location may be located within a plurality of pictures at the same location.

[0253] In the step of determining a block at the same location, a prediction mode is checked for each of a plurality of surrounding blocks spatially adjacent to the current block, and the surrounding block is encoded in an inter-prediction mode (first condition), and if the reference picture of the surrounding block is the same as the picture at the same location of the current block (second condition), the block at the location pointed to by the motion vector of the surrounding block within the picture at the same location can be set as the block at the same location.

[0254] Here, if the above first and second conditions are satisfied for multiple surrounding blocks, multiple blocks at the same location can be set for the current block.

[0255] For example, a plurality of surrounding blocks may include a left surrounding block (A1), a lower left surrounding block (A0), an upper surrounding block (B1), a upper right surrounding block (B0), and a upper left surrounding block (B2), as illustrated in FIG. 5.

[0256] In this case, whether the above first and second conditions are satisfied is checked in the order of the left surrounding block (A1), the lower left surrounding block (A0), the upper surrounding block (B1), the upper right surrounding block (B0), and the upper left surrounding block (B2), and one or more blocks at the same location can be set using one or more surrounding blocks that satisfy the first and second conditions.

[0257] Multiple surrounding blocks are not limited to the left surrounding block (A1), the lower left surrounding block (A0), the upper surrounding block (B1), the upper right surrounding block (B0), and the upper left surrounding block (B2), and may include surrounding blocks at various locations spatially adjacent to the current block.

[0258] For a neighbor block encoded in a unidirectional inter-prediction mode, the first and second conditions above are checked for the neighbor block, and if both conditions are satisfied, one same-location block is set, otherwise, no same-location block is set.

[0259] Meanwhile, for a surrounding block encoded in a bidirectional inter-prediction mode, the first and second conditions above are checked for the surrounding block, and depending on the result of the check in each direction, no blocks at the same location are set, or one or two blocks at the same location may be set.

[0260] As another example, for a neighbor block encoded in a bidirectional inter-prediction mode, if the neighbor block does not satisfy the first and second conditions above for both directions, no same location block is set, and if it satisfies the first and second conditions above for one of the two directions, one same location block may be set.

[0261] For example, referring to FIG. 8, if the surrounding block (A1) of the current block (curr_blk) is encoded in a bidirectional inter-prediction mode, and the reference picture (ref_pic_L0) of the reference picture list 0 (L0) for the surrounding block (A1) is the same as the same location picture (col_pic) of the current block (curr_blk), then the same location block (col_blk) can be set using a motion vector (indicated by a solid line) in the direction of the reference picture list 0 (L0).

[0262] Meanwhile, if the reference picture (ref_pic_L1) of the reference picture list 1 (L1) for the surrounding block (A1) is not the same as the same location picture (col_pic) of the current block (curr_blk), the same location block (col_blk) may not be set for the direction of the reference picture list 1 (L1).

[0263] Meanwhile, if the surrounding block satisfies the first and second conditions above for both directions, one block at the same location can be set using the motion vector of either of the two directions of motion vectors for the surrounding block.

[0264] In this case, as a criterion for selecting one of the two directions, the BCW (Bi-prediction with CU-level Weights) value, which represents the weights for the bidirectional prediction samples (or signals), can be used.

[0265] For example, depending on the BCW value for the surrounding blocks, the block at the same location in the direction with the higher weight between the two directions may be selected.

[0266] Meanwhile, if the two directions have the same weight, one specific direction of the two directions may be selected to set one block at the same location, or both directions may be selected to set two blocks at the same location.

[0267] Here, the BCW value may be specified as, for example, 3:5, 4:4, or 5:3, but the present invention is not limited thereto.

[0268] As another example, if the surrounding block satisfies the first and second conditions above for both directions, two blocks at the same location corresponding to each of the two directions may be set for the surrounding block.

[0269] When two blocks in the same location are set for a single surrounding block, the order in which they are selected as the final blocks in the same location can be determined based on the BCW value.

[0270] For example, among two blocks at the same location, the block at the same location with the larger weight value may have a higher order so that it is selected first, and the block at the same location with the smaller weight value may have a lower order so that it is selected later.

[0271] As another example, if the surrounding blocks satisfy the first and second conditions above for both directions, an evaluation using Template Matching (TM) may be performed.

[0272] Specifically, among the two identical location blocks corresponding to each of the two directions, the identical location block with the lower TM cost can be set as the identical location block for the corresponding surrounding block.

[0273] Referring to FIG. 9, the error value between the template area (curr_blk_template) already restored around the current block and the template area (col_blk_L0_template) of the corresponding L0 direction same location block (col_blk_L0) is compared with the error value between the template area (curr_blk_template) of the current block and the template area (col_blk_L1_template) of the corresponding L1 direction same location block (col_blk_L1), and the same location block in the direction with a lower TM cost (i.e., a smaller error value) can be selected.

[0274] For example, among the two directional template regions (col_blk_L0_template, col_blk_L1_template), if the template region in the L1 direction (col_blk_L1_template) has a lower TM cost (i.e., a smaller error value with the template region of the current block (curr_blk_template)), the same location block in the L1 direction (col_blk_L1) can be selected as the same location block for the surrounding block (A1).

[0275] Meanwhile, two identical location blocks corresponding to each of the two directions can both be set as identical location blocks for the corresponding surrounding blocks, and in this case, the order in which they are selected as the final identical location blocks can be determined according to the TM cost.

[0276] For example, among the two directional template regions (col_blk_L0_template, col_blk_L1_template), if the template region in the L1 direction (col_blk_L1_template) has a lower TM cost (i.e., a smaller error value with the current block's template region (curr_blk_template)), the L1 direction block at the same location (col_blk_L1) may have a higher order (e.g., lower index) so that it is selected first, and the L0 direction block at the same location (col_blk_L0) may have a lower order (e.g., higher index) so that it is selected later.

[0277] In this specification, the cost function for TM may be calculated using at least one of SAD (Sum of Absolute Differences), SATD (Sum of Absolute Transformed Differences), and MSE (Mean Squared Error), but the present invention is not limited thereto.

[0278] In the embodiments described above, for example, one block at the same location as the surrounding block (A1) is selected for the surrounding block (A1), and no block at the same location is selected for the surrounding block (A0), two blocks at the same location are selected for the surrounding block (B1), no block at the same location is selected for the surrounding block (B0), and one block at the same location is selected for the surrounding block (B2), so that a total of four blocks at the same location can be set for the current block.

[0279] To this end, a candidate list for blocks at the same location may be configured to include multiple blocks at the same location set for multiple surrounding blocks, and a predetermined number (e.g., 4) of blocks at the same location may be set for the current block according to the order within the candidate list.

[0280] Meanwhile, the order of blocks at the same location within the candidate list can be reordered according to the TM cost as described above.

[0281] Specifically, the order (or position) of blocks at the same location within the candidate list can be adjusted so that blocks with lower TM costs have a higher order (e.g., lower index) and blocks with higher TM costs have a lower order (e.g., higher index).

[0282] Referring to FIG. 10, there may be a template area (curr_blk_template) of the current block, a template area (col_blk_A1_template) of a block at the same location (col_blk_A1) set for a neighboring block (A1), and a template area (col_blk_B1_template) of a block at the same location (col_blk_B1) set for a neighboring block (B1).

[0283] For example, among the two template regions (col_blk_A1_template, col_blk_B1_template), if the template region (col_blk_A1_template) of the same location block (col_blk_A1) for the neighboring block (A1) has a lower TM cost (i.e., a smaller error value with the current block's template region (curr_blk_template)), then in the candidate list, the same location block (col_blk_A1) for the neighboring block (A1) may have a higher order (e.g., lower index) than the same location block (col_blk_B1) for the neighboring block (B1).

[0284] In another embodiment, multiple blocks at the same location set for multiple surrounding blocks can be grouped into two or more candidate lists.

[0285] Specifically, blocks at the same location within the first collocated picture are included in the first candidates list, and blocks at the same location within the second collocated picture may be included in the second candidates list.

[0286] For example, among the surrounding blocks adjacent to the current block, for a surrounding block encoded in an inter-prediction mode and whose reference picture is identical to the first same-location picture of the current block, the same-location block may be added to the first candidate list.

[0287] Meanwhile, for surrounding blocks encoded in inter-prediction mode where the reference picture is identical to the second identical location picture of the current block, the identical location block may be added to the second candidate list.

[0288] In this case, for surrounding blocks encoded in a bidirectional inter-prediction mode, blocks at the same location for each direction may be added to the first candidate list and / or the second candidate list.

[0289] Meanwhile, the order of blocks at the same location within each candidate list can be reordered according to the TM cost as described above.

[0290] Specifically, the order (or position) of blocks at the same location within each candidate list can be adjusted so that blocks with lower TM costs have a higher order (e.g., lower index) and blocks with higher TM costs have a lower order (e.g., higher index).

[0291] Referring to FIG. 11, if both the surrounding block (A1) and the surrounding block (B1) are encoded in an inter-prediction mode and the reference picture is the same as the first same-location picture (col_pic1) of the current block, the same-location block (col_blk_A1) set for the surrounding block (A1) and the same-location block (col_blk_B1) set for the surrounding block (B1) can be added to the first candidate list.

[0292] And, for example, among the template regions (col_blk_A1_template, col_blk_B1) of each of the two identical blocks (col_blk_A1, col_blk_B1), if the template region (col_blk_A1_template) of the identical block (col_blk_A1) for the neighboring block (A1) has a lower TM cost (i.e., a smaller error value with the current block's template region (curr_blk_template)), then in the first candidate list, the identical block (col_blk_A1) for the neighboring block (A1) may have a higher order (e.g., lower index) than the identical block (col_blk_B1) for the neighboring block (B1).

[0293] Meanwhile, if up to 4 sbTMVP candidates are set, the top 2 identical location blocks within the first candidate list for the first identical location picture may be set for the current block, and the top 2 identical location blocks within the second candidate list for the second identical location picture may be set for the current block.

[0294] As another example, without distinguishing between the first and second identical location pictures, based on the TM cost, the four identical location blocks with the smallest TM cost within the first and second candidate lists may be set for the current block.

[0295] For example, the top 1 same-location block in the first candidate list for the first same-location picture may be set for the current block, and the top 3 same-location blocks in the second candidate list for the second same-location picture may be set for the current block.

[0296] Although the above description describes an embodiment of the present invention using the case where the same position picture is fixed as an example, the present invention is not limited thereto, and below, embodiments regarding a method for determining the same position block when the same position picture is variably changed will be described.

[0297] In this embodiment, a picture at the same location can be set as a reference picture of a neighboring block adjacent to the current block, for example, if the reference picture of a neighboring block is the first reference picture in the reference picture list 0 (L0), that reference picture can be set as a block at the same location for the current block.

[0298] In this case, the corresponding surrounding block must be a block encoded in an inter-prediction mode, and the process of checking the prediction mode for multiple surrounding blocks spatially adjacent to the current block may be as described above.

[0299] For example, one collocated set consisting of a picture at the same location and a block at the same location is set using a surrounding block (A1), and no collocated set is set for the surrounding block (A0), two collocated sets are set using a surrounding block (B1), no collocated set is set for the surrounding block (B0), and one collocated set is set using a surrounding block (B2), so that a total of four collocated sets consisting of a picture at the same location and a block at the same location can be set.

[0300]

[0301] In the step of obtaining movement information of blocks at the same location, movement information for each of multiple blocks at the same location is obtained, and the obtained movement information can be added as sbTMVP candidates to the candidate list for sub-block-based merge mode.

[0302] To this end, a prediction mode for a collocated sub-block within a collocated block is identified, and if the collocated sub-block is encoded in an inter-prediction mode, movement information of the collocated sub-block can be obtained.

[0303] And the movement information of the acquired sub-block at the same location can be used as basic movement information for the corresponding block at the same location.

[0304] Here, the same-location sub-block may be set as a predetermined area (sub-block or unit) corresponding to the center (or upper-left) position within the same-location block, but the present invention is not limited thereto and may be an area corresponding to a specific position within the same-location block.

[0305] Meanwhile, the size of a sub-block at the same location may be the same as the size unit stored in a motion information storage (e.g., MV memory or buffer) so that the motion information of the block is used as a motion vector for another block later.

[0306] For example, if motion information is stored in MV memory in units of 8x8, the size of the sub-block at the same location may be 8x8, and if motion information is stored in MV memory in units of 4x4, the size of the sub-block at the same location may be 4x4.

[0307] In another embodiment, the prediction mode of each of all sub-blocks within the same location block is identified, and if a predetermined number or more of sub-blocks are encoded in an inter-prediction mode, the motion information of one of the sub-blocks encoded in an inter-prediction mode can be set as the basic motion information for the corresponding same location block.

[0308] In another embodiment, specific movement information may be set as default movement information for the same location block without checking the prediction mode for the sub-block within the same location block.

[0309] In this case, the reference picture according to specific motion information is set to the 0th reference picture in the reference picture list (L0 or L1) of the slice where the same location block is located, or the same location picture of the slice where the same location block is located, and the motion vector can be set to a zero motion vector (zero MV) or a motion vector value representing the slice.

[0310] In another embodiment, if the prediction mode of each of all sub-blocks within the same location block is identified and not all sub-blocks are encoded in an inter-prediction mode, the specific motion information described above may be set as the basic motion information for the corresponding same location block.

[0311]

[0312] In the step of obtaining movement information of sub-blocks, movement information for each sub-block within the same location block can be obtained.

[0313] Specifically, the prediction mode of each sub-block within the same location block is identified, and if the corresponding sub-block is encoded in an inter-prediction mode, movement information of the corresponding sub-block can be obtained.

[0314] Here, the movement information of a sub-block may refer to movement information within MV memory that covers the central location sample of the corresponding sub-block.

[0315] Meanwhile, if the corresponding sub-block is not encoded in inter-prediction mode, the basic motion information for the same location block may be used as motion information for the corresponding sub-block.

[0316] In another embodiment, if the corresponding sub-block is not encoded in an inter-prediction mode, the motion information of a surrounding sub-block spatially adjacent to the corresponding sub-block, rather than the basic motion information for the same location block, may be used as motion information for the corresponding sub-block.

[0317] As another embodiment, a refinement operation using TM can be performed on the movement information of a sub-block within the same location block obtained as above.

[0318] Specifically, the motion vector of a sub-block within the same location block can be corrected to a motion vector corresponding to the template area with the lowest TM cost within a certain area of ​​the reference picture for the sub-block.

[0319] Referring to Fig. 12, error values ​​are calculated between the template region (sub-block_template) that has already been restored around the sub-block within the same location block (col_blk) and the template regions (ref_sub-block_templates) within a certain area (R) of the reference picture (ref_pic) for the sub-block, so that the template region with the lowest TM cost (i.e., the smallest error value) within the certain area (R) of the reference picture (ref_pic) can be selected.

[0320] And, with the position indicated by the motion vector (indicated by a solid line) of the corresponding sub-block as the initial position, an offset value representing the displacement from that initial position to the position of the template area (indicated by a dotted line) with the lowest TM cost can be determined.

[0321] Accordingly, the offset value determined as above is added to the motion vector of the corresponding sub-block (indicated by a solid line), and a corrected motion vector (indicated by a dotted line) can be obtained.

[0322] Meanwhile, the offset value determined as above is shared among all sub-blocks within the same location block (col_blk) and can be added to the motion vector of each sub-block within the same location block (col_blk).

[0323]

[0324] In the step of scaling motion vectors (MV scaling), a motion vector scaling operation may be performed to convert the motion vector values ​​of each sub-block within the same location block into motion vector values ​​corresponding to each sub-block within the current block.

[0325] In FIG. 13, the current picture (curr_pic) refers to the slice to which the current block (curr_blk), which is the block to be decoded, belongs, and the same location picture (col_pic) may refer to the slice to which the same location block (col_blk) belongs.

[0326] Additionally, the current block reference picture (curr_ref_pic) refers to the reference picture of the current block (curr_blk), and the same location block reference picture (col_ref_pic) may refer to the reference picture of the same location block (col_blk) or the same location sub-block (collocated sub-block).

[0327] Here, the reference picture (curr_ref_pic) of the current block (curr_blk) may be the 0th reference picture in the list of reference pictures in the same direction as the picture at the same location (col_pic).

[0328] As another example, the reference picture (curr_ref_pic) of the current block (curr_blk) can be the picture (col_pic) at the same location in the slice to which the current block (curr_blk) belongs.

[0329] Specifically, if an sbTMVP candidate is included in the candidate list for sub-block based merge mode or general merge mode, the reference picture (curr_ref_pic) of the current block (curr_blk) may be set to one of the reference pictures in the list of reference pictures of the slice to which the current block (curr_blk) belongs.

[0330] Meanwhile, if an sbTMVP candidate is included in the AMVP candidate list, information about the reference picture (curr_ref_pic) of the current block (curr_blk) can be signaled.

[0331] Referring to FIG. 13, by using the movement information of the surrounding blocks (nb_blk) of the current block (curr_blk) within the current picture (curr_pic), the same location block (col_blk) within the same location picture (col_pic) is determined, and the movement information of the same location block (col_blk) or the same location sub-block (collocated sub-block) within the same location block can be obtained.

[0332] And through the motion vector scaling (MV scaling) operation, the motion vectors of the same-location block (col_blk) or the same-location sub-blocks within the same-location block (collocated sub-block) can be converted into the motion vectors (indicated by dotted lines) corresponding to the current block (curr_blk).

[0333] In this case, the scaling factor can be calculated by the ratio of the difference in POC between the same location picture (col_pic) and the same location block reference picture (col_ref_pic) (col_Poc_diff) and the difference in POC between the current picture (curr_pic) and the current block reference picture (curr_ref_pic), as shown in Equation 2 above.

[0334]

[0335] At least some of the processes for finding sbTMVP candidates as described above can also be used in the process for finding TMVP candidates.

[0336] That is, the embodiments of the present invention as described above are not limited to sbTMVP, and the remaining steps, excluding the "step of obtaining movement information of a sub-block" during the process of obtaining sbTMVP candidates, may also be performed in the process of obtaining TMVP candidates.

[0337] For example, a TMVP candidate can be derived by performing the steps of “determining a picture at the same location,” “determining a block at the same location,” “obtaining motion information of a block at the same location,” and “scaling a motion vector” among the sbTMVP candidate derivation method according to an embodiment of the present invention.

[0338]

[0339] In addition, the sbTMVP candidate for the sub-block based merge mode described above can also be used in the MMVD (Merge with Motion Vector Difference) mode.

[0340] For example, motion vector candidates for MMVD mode can be obtained by performing the remaining steps, excluding the "step of determining the same location block" during the process of obtaining sbTMVP candidates.

[0341] In MMVD mode, motion information of the block to be decoded is signaled as a merge candidate index and a Motion Vector Difference (MVD), and the MVD can represent offset values ​​of the horizontal and vertical axes that are added to the motion vector signaled as the merge candidate index.

[0342] In this case, the block at the location indicated by the motion vector in which the MVD is added to the motion vector signaled as a merge candidate index can be a collocated block according to an embodiment of the present invention.

[0343] That is, in sub-block based merge mode, blocks at the same location are set using movement information of surrounding blocks spatially adjacent to the block to be decoded (current block), and in MMVD mode, blocks at the same location can be determined by adding or subtracting an offset value according to MVD to a movement vector signaled by the movement information of surrounding blocks.

[0344]

[0345] And the sbTMVP candidate for the sub-block based merge mode described above can also be used in AMVP mode.

[0346] For example, motion vector candidates for AMVP mode can be obtained by performing the remaining steps, excluding the "step of determining the same-location picture" and the "step of determining the same-location block," during the process of obtaining sbTMVP candidates.

[0347] In AMVP mode, information about a reference picture and a motion vector is signaled as motion information for a block to be decoded, and the information about the motion vector may include an MVP (Motion Vector Predictor) and an MVD.

[0348] Here, MVP represents the reference position for the motion vector, and MVD may represent the horizontal and vertical axis offset values ​​added to MVP.

[0349] In this case, the reference picture for the block to be decoded (current block) is set to a collocated picture, and the location pointed to by the motion vector value can be set to a collocated block.

[0350] That is, in sub-block based merge mode, blocks at the same location are set using movement information of surrounding blocks spatially adjacent to the block to be decoded (current block), and in AMVP mode, blocks at the same location can be determined by movement information (MVP and MVD) signaled to the block to be decoded (current block).

[0351]

[0352] The inter prediction method described above can be performed by an image encoding device and an image decoding device according to an embodiment of the present invention, and, for example, by an inter prediction unit (165) of a video encoder (100) shown in FIG. 1 or an inter prediction unit (265) of a video decoder (200) shown in FIG. 3.

[0353]

[0354] The methods described above in this specification may be performed through a processor of a video encoder or a video decoder. Additionally, the encoder may generate a bitstream that is decoded by a video signal processing method, and the bitstream generated by the encoder may be stored in a computer-readable non-transient storage medium (recording medium).

[0355] The embodiments of the present invention described above may be implemented through various means. For example, the embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

[0356] Some embodiments may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media.

[0357] Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically include other data of modulated data signals, such as computer-readable instructions, data structures, or program modules, or other transmission mechanisms, and include any information transmission media.

[0358] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. In a method for performing sub-block-based inter-prediction, A step of determining a collocated picture for the current block; A step of determining a collocated block within the above-mentioned picture at the same location; A step of obtaining movement information of the same block determined above; A step of obtaining movement information for each sub-block within the same location block; and It includes a step of scaling the motion vector; and An inter prediction method characterized by the fact that a plurality of blocks at the same location can be set for the current block.

2. In paragraph 1, the scaled motion vector is Inter prediction method included in the candidate list for sub-block based merge mode.

3. In Paragraph 1, An inter-prediction method in which two identical position pictures corresponding to both directions can be set for B slice.

4. In paragraph 1, the above-mentioned picture at the same location is An inter-prediction method that is adaptively set according to a reference picture of a surrounding block spatially adjacent to the current block.

5. In Paragraph 1, An inter prediction method in which, when surrounding blocks spatially adjacent to the current block are encoded in a bidirectional inter prediction mode, two pictures of the same location corresponding to both directions can be set based on movement information of the surrounding blocks.

6. In Paragraph 1, An inter-prediction method further comprising the step of performing an evaluation operation using Template Matching (TM) for the same location blocks determined above.

7. In Paragraph 1, An inter-prediction method further comprising the step of performing a refinement operation using Template Matching (TM) on the movement information of the sub-block obtained above.

8. An image encoding device that performs the method of any one of paragraphs 1 to 7.

9. An image decoding device that performs the method of any one of paragraphs 1 through 7.

Images