Video encoding / decoding method and device
The method addresses inefficiencies in high-resolution and stereoscopic video compression by deriving and correcting prediction vectors for sub-blocks, enhancing encoding and decoding efficiency.
Patent Information
- Application Number
- PCT/KR2026/096708
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
Existing video compression technologies struggle to efficiently encode and decode high-resolution and stereoscopic video content, particularly in handling prediction vectors and block vectors for accurate prediction.
A method and apparatus for deriving and correcting prediction vectors for each sub-block of a current block, utilizing motion vectors or block vectors, and performing predictions based on block vector-based, average, or weighted average modes, with corrections in units of integer pixels or subpixels, and determining prediction based on block information and geometric partitioning.
Enhances the efficiency of video encoding and decoding by improving prediction accuracy and reducing bit usage, particularly for high-resolution and stereoscopic video content.
Smart Images

Figure KR2026096708_12112026_PF_FP_ABST
Abstract
Description
Video encoding / decoding method and device
[0001] The present invention relates to a video encoding / decoding method and apparatus.
[0002] Market demand for high-resolution video is increasing, and consequently, there is a need for technology that can efficiently compress high-resolution video. In response to these market demands, the Moving Picture Expert Group (MPEG) of ISO / IEC and the Video Coding Expert Group (VCEG) of ITU-T jointly formed the Joint Collaborative Team on Video Coding (JCT-VC), completed the development of the High Efficiency Video Coding (HEVC) video compression standard in January 2013, and have been actively conducting research and development on next-generation compression standards.
[0003] Meanwhile, along with the increasing demand for high-resolution video, the demand for stereoscopic video content as a new video service is also rising. Discussions are underway regarding video compression technologies to effectively provide high-resolution and ultra-high-resolution stereoscopic video content.
[0004] The present disclosure aims to provide a method and apparatus for deriving a prediction vector for each sub-block for a current block.
[0005] The present disclosure aims to provide a method and apparatus for generating a predicted block of a current block based on a prediction vector for each sub-block of a corrected current block.
[0006] The present disclosure aims to provide a method and apparatus for correcting the resolution of a prediction vector.
[0007] The present disclosure aims to provide a method and apparatus for deriving a prediction vector based on a block vector of a reference block.
[0008] The image decoding method and apparatus according to the present disclosure may derive a prediction vector for each sub-block for a current block, correct the prediction vector, and perform a prediction for the current block based on the corrected prediction vector. Here, the prediction vector includes at least one of a motion vector or a block vector, and the prediction for the current block may be performed based on at least one of a block vector-based mode, an average mode, or a weighted average mode.
[0009] In the image decoding method and apparatus according to the present disclosure, whether to perform the prediction may be determined based on at least one of the current block, block information of the surrounding blocks of the current block, or the prediction method of the surrounding blocks, and the block information may include at least one of width, height, area, or aspect ratio.
[0010] In the image decoding method and apparatus according to the present disclosure, the number of predicted directions may be determined based on at least one of the height or width of the current block.
[0011] In the image decoding method and apparatus according to the present disclosure, the correction can be performed on the vector resolution of the prediction vector.
[0012] In the image decoding method and apparatus according to the present disclosure, the prediction vector may be derived based on the block vector of a reference block, and the correction may be performed in units of integer pixels or subpixels.
[0013] In the image decoding method and apparatus according to the present disclosure, the prediction vector may be derived based on the weighted average of the prediction vectors of the surrounding blocks of the current block.
[0014] In the image decoding method and apparatus according to the present disclosure, the peripheral block may be a block coded based on at least one of prediction vector-based prediction, intra prediction, or inter prediction.
[0015] In the image decoding method and apparatus according to the present disclosure, the prediction vector may be derived based on a vector pointing to a first reference image, and the corrected prediction vector may be a vector pointing only to the first reference image.
[0016] In the image decoding method and apparatus according to the present disclosure, the prediction vector may be derived based on a vector pointing to a plurality of reference images.
[0017] In the image decoding method and apparatus according to the present disclosure, the correction may be performed according to at least one of a sub-block unit, a unit larger than a sub-block, a unit smaller than a sub-block, or a pixel unit.
[0018] In the image decoding method and apparatus according to the present disclosure, when the current block is encoded based on a geometric partitioning mode, the prediction vector-based prediction can be performed for each partitioned region partitioned based on the geometric partitioning mode.
[0019] The image encoding method and apparatus according to the present disclosure may derive a prediction vector for each sub-block for a current block, correct the prediction vector, and perform a prediction for the current block based on the corrected prediction vector. Here, the prediction vector includes at least one of a motion vector or a block vector, and the prediction for the current block may be performed based on at least one of a block vector-based mode, an average mode, or a weighted average mode.
[0020] A computer-readable recording medium according to the present disclosure can store a bitstream encoded by the image encoding method.
[0021] The bitstream transmission method and apparatus according to the present disclosure can transmit a bitstream generated by the image encoding method.
[0022] According to the present disclosure, intra / inter prediction for the current block can be performed based on a prediction vector.
[0023] According to the present disclosure, the prediction vector used for the prediction of the current block may be a motion vector or a block vector.
[0024] According to the present disclosure, whether to perform prediction based on a prediction vector can be determined based on block information of the current block or surrounding blocks.
[0025] According to the present disclosure, block vectors can be corrected in units of integer pixels or subpixels.
[0026] According to the present disclosure, motion vectors can be corrected to point only to the same reference picture.
[0027] According to the present disclosure, the resolution of the prediction vector can be corrected.
[0028] According to the present disclosure, intra / inter prediction for the current block can be performed based on a corrected prediction vector.
[0029] According to the present disclosure, the current block can be divided into sub-blocks, and prediction vector-based prediction can be performed for each divided sub-block.
[0030] FIG. 1 is a block diagram showing an image encoding device according to the present disclosure.
[0031] FIG. 2 is a block diagram showing an image decoding device according to the present disclosure.
[0032] FIG. 3 illustrates an image decoding method utilizing motion vectors and / or block vectors according to the present disclosure.
[0033] Figure 4 shows the sub-blocks of the current block and surrounding blocks.
[0034] FIG. 5 is a block diagram illustrating a method of encoding an image based on an image encoding method performed in an image encoding device according to the present disclosure.
[0035] Embodiments of the present invention are described in detail with reference to the drawings attached to this specification so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.
[0036] Throughout this specification, when a part is described as being 'connected' to another part, this includes not only cases where they are directly connected, but also cases where they are electrically connected with other elements in between.
[0037] Furthermore, throughout this specification, when a part is described as 'comprising' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0038] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0039] Additionally, in the embodiments relating to the device and method described herein, some components of the device or some steps of the method may be omitted. Also, the order of some components of the device or some steps of the method may be changed. Additionally, other components or other steps may be inserted into some components of the device or some steps of the method.
[0040] In addition, some components or steps of the first embodiment of the present invention may be added to or replace some components or steps of the second embodiment of the present invention.
[0041] Furthermore, the components shown in the embodiments of the present invention are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, for convenience of explanation, each component is described by listing it as a separate component, and at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function. Such integrated and separated embodiments of each component are also included within the scope of the present invention as long as they do not deviate from the essence of the invention.
[0042] In this specification, "Current Block" may be a specific unit block subject to encoding or decoding, or it may be understood as a higher-level concept encompassing a set of multiple sub-blocks formed by dividing the said block into any size or shape. In various embodiments of the present invention, each step described as being performed on the current block may be applied individually or integrally to each of the multiple sub-blocks included within the current block in actual implementation. Even if the explicit description in this specification targets the current block, this should be interpreted to include each step of the present invention performed at the level of each sub-block, which is a sub-unit of the current block, unless otherwise specifically stated.
[0043] In this specification, a block may be expressed in various ways such as a unit, region, unit, partition, etc., and a sample may be expressed in various ways such as a pixel, pel, pixel, etc. Pixel, pel, and pixel may have the same meaning.
[0044] Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. In describing the present invention, redundant descriptions of identical components are omitted.
[0045] FIG. 1 is a block diagram showing an image encoding device according to the present disclosure.
[0046] Referring to FIG. 1, the image encoding device (100) may include a picture splitting unit (110), a prediction unit (120, 125), a conversion unit (130), a quantization unit (135), a reordering unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse conversion unit (145), a filter unit (150), and a memory (155).
[0047] The picture splitting unit (110) can split the input picture into at least one processing unit. At this time, the processing unit may be a Prediction Unit (PU), a Transform Unit (TU), or a Coding Unit (CU). In the embodiments of the present disclosure below, the term "coding unit" may be used to mean a unit that performs coding or a unit that performs decoding.
[0048] A prediction unit may be divided into at least one square or rectangle of the same size within a single coding unit, or it may be divided such that one of the prediction units within a single coding unit has a different shape and / or size from another prediction unit. When generating a prediction unit that performs intra prediction based on a coding unit, if it is not the minimum coding unit, intra prediction can be performed without dividing into multiple prediction units NxN.
[0049] The prediction unit (120, 125) may include an inter prediction unit (120) that performs inter prediction or inter prediction, and an intra prediction unit (125) that performs intra prediction or intra prediction. It may determine whether to use inter prediction or perform intra prediction for a prediction unit, and determine specific information according to each prediction method (e.g., intra prediction mode, motion vector, reference picture, etc.). The residual value (residual block) between the generated prediction block and the original block may be input to the conversion unit (130). In addition, the prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value in the entropy encoding unit (165) and transmitted to the decoder.
[0050] The inter prediction unit (120) may predict a prediction unit based on information of at least one of the previous or subsequent pictures of the current picture, and in some cases, may predict a prediction unit based on information of a partially encoded area within the current picture. The inter prediction unit (120) may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.
[0051] In the reference picture interpolation unit, reference picture information is received from memory (155), and pixel information of integer pixels or less can be generated from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter with different filter coefficients can be used to generate pixel information of integer pixels or less in 1 / 4 pixel units. In the case of chrominance signals, a DCT-based 4-tap interpolation filter with different filter coefficients can be used to generate pixel information of integer pixels or less in 1 / 8 pixel units.
[0052] The motion prediction unit can perform motion prediction based on a reference picture interpolated by the reference picture interpolation unit. Various methods, such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm), can be used to calculate motion vectors. Based on the interpolated pixels, the motion vector can have motion vector values in units of 1 / 2 or 1 / 4 pixels. The motion prediction unit can predict the current prediction unit by using different motion prediction methods. Various methods, such as Skip Mode, Merge Mode, AMVP Mode, Intra Block Copy Mode, Affine Mode, and Template Matching Mode, can be used as motion prediction methods.
[0053] The intra prediction unit (125) can generate a prediction unit based on reference pixel information around the current block, which is pixel information within the current picture. If the surrounding block of the current prediction unit is a block that has performed inter prediction, and the reference pixel is a pixel that has performed inter prediction, the reference pixel included in the block that has performed inter prediction can be replaced with the reference pixel information of the surrounding intra prediction block. That is, if the reference pixel is not available, the unavailable reference pixel information can be replaced with at least one of the available reference pixels.
[0054] Additionally, a residual block can be generated that includes residual value information, which is the difference between the prediction unit that performed the prediction based on the prediction unit generated in the prediction unit (120, 125) and the original block of the prediction unit. The generated residual block can be input to the conversion unit (130).
[0055] In the transformation unit (130), the residual block containing residual value information of the prediction unit generated through the original block and the prediction unit (120, 125) can be transformed using a transformation method such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), or KLT. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on the intra-prediction mode information of the prediction unit used to generate the residual block.
[0056] The quantization unit (135) can quantize the values converted into the frequency domain in the conversion unit (130). The quantization coefficient may vary depending on the block or the importance of the image. The values produced by the quantization unit (135) may be provided to the inverse quantization unit (140) and the reordering unit (160).
[0057] The reordering unit (160) can perform reordering of coefficient values for quantized residual values.
[0058] The reordering unit (160) can convert two-dimensional block-shaped coefficients into one-dimensional vector forms through a coefficient scanning method. For example, the reordering unit (160) can convert the coefficients from DC to high-frequency range coefficients into one-dimensional vector forms by scanning using a Zig-Zag Scan method. Depending on the size of the conversion unit and the intra-prediction mode, a vertical scan that scans two-dimensional block-shaped coefficients in the column direction or a horizontal scan that scans two-dimensional block-shaped coefficients in the row direction may be used instead of the Zig-Zag Scan. That is, depending on the size of the conversion unit and the intra-prediction mode, it can be determined whether to use a Zig-Zag Scan, a vertical scan, or a horizontal scan.
[0059] The entropy encoding unit (165) can perform entropy encoding based on values calculated by the reordering unit (160). Entropy encoding can use various encoding methods, such as, for example, Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding). In this regard, the entropy encoding unit (165) can encode residual value coefficient information of the encoding unit from the reordering unit (160) and the prediction unit (120, 125). Additionally, according to the present disclosure, it is possible to signal and transmit information indicating that motion information is derived and used from the decoder side, and information regarding the technique used to derive motion information.
[0060] In the inverse quantization unit (140) and inverse transformation unit (145), the values quantized in the quantization unit (135) are inverse quantized, and the values transformed in the transformation unit (130) are inverse transformed. The residual value generated in the inverse quantization unit (140) and inverse transformation unit (145) can be combined with the prediction unit predicted through the motion estimation unit, motion compensation unit, and intra prediction unit included in the prediction unit (120, 125) to generate a reconstructed block.
[0061] The filter unit (150) may include at least one of a deblocking filter, an offset correction unit, and an Adaptive Loop Filter (ALF). The deblocking filter can remove block distortion caused by boundaries between blocks in the restored picture. The offset correction unit can correct the offset from the original image on a pixel-by-pixel basis for the image that has undergone deblocking. To perform offset correction for a specific picture, a method may be used in which pixels included in the image are divided into a certain number of regions, the region to be offset is determined, and the offset is applied to that region, or a method may be used in which the offset is applied by considering the edge information of each pixel. Adaptive Loop Filtering (ALF) may be performed based on a value obtained by comparing the filtered restored image with the original image. After dividing the pixels included in the image into a predetermined group, a filter to be applied to that group is determined, and filtering may be performed differentially for each group.
[0062] The memory (155) can store a restored block or picture produced through the filter unit (150), and the stored restored block or picture can be provided to the prediction unit (120, 125) when performing inter-prediction.
[0063] FIG. 2 is a block diagram showing an image decoding device according to the present disclosure.
[0064] Referring to FIG. 2, the image decoding device (200) may include an entropy decoding unit (210), a reordering unit (215), an inverse quantization unit (220), an inverse transformation unit (225), a prediction unit (230, 235), a filter unit (240), and a memory (245).
[0065] When a video bitstream is input to a video encoding device, the input bitstream can be decoded by the reverse procedure of the video encoding device.
[0066] The entropy decoding unit (210) can perform entropy decoding in the opposite procedure to that which the entropy encoding unit of the image encoder performed. For example, various methods such as Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) can be applied in correspondence with the method performed in the image encoder.
[0067] The entropy decoding unit (210) can decode information related to intra-prediction and inter-prediction performed in the encoder.
[0068] The reordering unit (215) can perform reordering based on the method of reordering the entropy-decoded bitstream in the encoding unit in the entropy decoding unit (210). The coefficients expressed in the form of a one-dimensional vector can be reordered by restoring them to the form of two-dimensional block coefficients.
[0069] The inverse quantization unit (220) can perform inverse quantization based on the coefficient values of the rearranged block and the quantization parameters provided by the encoder.
[0070] The inverse transform unit (225) can perform inverse transforms, i.e., inverse DCT, inverse DST, and inverse KLT, on the transforms, i.e., DCT, DST, and KLT, performed by the transform unit on the quantization result performed by the image encoder. The inverse transform can be performed based on the transmission unit determined by the image encoder. In the inverse transform unit (225) of the image decoder, a transformation technique (e.g., DCT, DST, KLT) can be selectively performed according to multiple pieces of information such as a prediction method, the size of the current block, and the prediction direction.
[0071] The prediction unit (230, 235) can generate a prediction block based on the prediction block generation information provided by the entropy decoding unit (210) and the previously decoded block or picture information provided by the memory (245).
[0072] As described above, when performing intra prediction or intra prediction identical to the operation in the video encoder, if the size of the prediction unit and the size of the transform unit are the same, intra prediction for the prediction unit is performed based on the pixels to the left of the prediction unit, the pixels to the top left, and the pixels to the top; however, if the size of the prediction unit and the size of the transform unit are different when performing intra prediction, intra prediction can be performed using reference pixels based on the transform unit. Additionally, intra prediction using NxN partitioning only for the minimum encoding unit may also be used.
[0073] The prediction unit (230, 235) may include a prediction unit determination unit, an inter prediction unit, and an intra prediction unit. The prediction unit determination unit receives various information, such as prediction unit information input from the entropy decoding unit (210), prediction mode information of the intra prediction method, and motion prediction related information of the inter prediction method, distinguishes the prediction unit in the current encoding unit, and determines whether the prediction unit performs inter prediction or intra prediction. On the other hand, if the encoder (100) does not transmit motion prediction related information for the inter prediction, but instead transmits information indicating that motion information is derived and used from the decoder side and information regarding the technique used for deriving motion information, the prediction unit determination unit determines whether the inter prediction unit (230) performs prediction based on the information transmitted from the encoder (100).
[0074] The inter prediction unit (230) can perform inter prediction for the current prediction unit based on information included in at least one picture, either a previous picture or a subsequent picture, of the current picture containing the current prediction unit, using information required for inter prediction of the current prediction unit provided by the video encoder. To perform inter prediction, based on the encoding unit, it can determine whether the motion prediction method of the prediction unit included in the corresponding encoding unit is a Skip Mode, Merge Mode, AMVP Mode, Intra Block Copy Mode, Affine Mode, or Template Matching Mode.
[0075] The intra prediction unit (235) can generate a prediction block based on pixel information within the current picture. If the prediction unit is a prediction unit that has performed intra prediction, it can perform intra prediction based on the intra prediction mode information of the prediction unit provided by the image encoder.
[0076] The intra prediction unit (235) may include an Adaptive Intra Smoothing (AIS) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a part that performs filtering on the reference pixels of the current block, and can determine whether to apply the filter based on the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block using the prediction mode of the prediction unit and AIS filter information provided by the image encoder. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.
[0077] The reference pixel interpolation unit can generate a reference pixel of an integer value or less by interpolating the reference pixel when the prediction mode of the prediction unit is a prediction unit that performs intra-prediction based on the pixel value interpolated from the reference pixel. If the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is DC mode.
[0078] The restored block or picture may be provided to a filter unit (240). The filter unit (240) may include a deblocking filter, an offset correction unit, and an ALF.
[0079] Information regarding whether a deblocking filter has been applied to the corresponding block or picture can be received from the video encoding device, and if a deblocking filter has been applied, information regarding whether a strong filter or a weak filter has been applied. The deblocking filter of the video decoder receives information related to the deblocking filter provided by the video encoder and can perform deblocking filtering on the corresponding block.
[0080] The offset correction unit can perform offset correction on the restored image based on the type of offset correction and offset value information applied to the image during encoding. ALF can be applied to the encoding unit based on information on whether ALF is applied and ALF coefficient information provided by the encoder. This ALF information can be provided included in a specific parameter set.
[0081] The memory (245) can store the restored picture or block so that it can be used as a reference picture or reference block, and can also provide the restored picture to the output unit.
[0082] In image coding, inter-prediction uses a technique that patches prediction blocks similar to the current block from the encoded / decoded reference picture to eliminate temporal redundancy. In contrast, intra-prediction generates prediction blocks based on reference samples. Meanwhile, block vector-based prediction is a technique used under the assumption that regions similar to the current block exist even within the same picture; it employs a method of patching regions similar to the current block from already encoded / decoded areas within the same picture.
[0083] Block vector-based prediction can be defined as a method in which the decoder generates an implicit block vector list to reduce the amount of bits used for block vector signaling, and generates a predicted block based on an index that specifies a block vector in the block vector list. Block vectors used prior to the current block can be stored and managed in a list, and this list can be utilized when block vector-based prediction is used in blocks that are subsequently encoded or decoded. Examples of block vector-based prediction may include Intra Block Copy (IBC) or Intra Template Matching Prediction (TMP).
[0084] The image decoding method and apparatus according to the present disclosure can generate predicted samples by utilizing motion vectors or block vectors for subblocks constituting the current block.
[0085] Block vector-based prediction, mean-based prediction, reference image-based prediction, and vector interpolation-based prediction according to the present disclosure may all be prediction vector-based prediction methods. Block vector-based prediction mode, mean-based prediction mode, reference image-based prediction mode, and vector interpolation-based prediction mode may all be prediction vector-based modes.
[0086] The prediction vector according to the present disclosure may be a motion vector or a block vector. A prediction vector may be derived to perform prediction vector-based prediction. In this case, the prediction vector may be derived after it is determined that prediction vector-based prediction will be performed.
[0087] FIG. 3 illustrates an image decoding method utilizing motion vectors and / or block vectors according to the present disclosure.
[0088] According to FIG. 3, a prediction vector used for predicting the current block can be derived for each sub-block (S300), and a prediction can be performed based on the prediction vector (S310). Each step (S300, S310) illustrated in FIG. 3 will be described below under the detailed description of each method.
[0089] An image encoder / decoder according to the present disclosure can signal screen content compression information. The decoder can determine whether to perform block vector prediction from the screen content compression information.
[0090] Whether to perform prediction vector-based prediction on the current block can be determined based on information of the current block or information of surrounding blocks. The information of the current block may include at least one of the width, height, area, or number of pixels within the block of the current block. The information of surrounding blocks may be a compression method for surrounding blocks.
[0091] For example, if the width is greater than A and the height is greater than B, motion vector prediction or block vector prediction can be performed. A and B can be any natural numbers.
[0092] Alternatively, prediction vector-based prediction can be performed only when the number of pixels within a block is greater than C. C can be any natural number.
[0093] Alternatively, prediction vector-based prediction can be performed only when the area of the current block is greater than D. D can be any natural number. Specifically, prediction vector-based prediction can be performed when the area of the current block is 32 or greater.
[0094] Meanwhile, by utilizing multiple contexts, the prediction direction index information used when performing prediction vector-based prediction for sub-blocks can be compressed. For example, the prediction direction index information may be included in a mode list. Additionally, in the above example, different contexts may be used for each case by distinguishing between when the area of the current block is greater than 32 and when it is less than 32. The context may be the average or weighted sum average of the directions of the block vectors of surrounding blocks. Alternatively, the context may be at least one of the width, height, area, or number of pixels within the current block. Alternatively, the context may be the context of surrounding blocks.
[0095] When performing intra prediction for the current block, the current block and surrounding blocks can each be divided into multiple sub-blocks to perform intra prediction. In this case, prediction vector-based prediction can be performed for each sub-block of the current block. The motion vectors of the surrounding block's sub-blocks can have various directions, such as horizontal, vertical, or diagonal directions. The motion vectors of the surrounding block's sub-blocks can be assigned to each sub-block of the current block. Prediction for each sub-block can be performed based on the motion vectors assigned in this manner.
[0096] Block vectors of neighboring blocks or sub-blocks of neighboring blocks can be derived. Block vectors derived from neighboring blocks or sub-blocks of neighboring blocks can be assigned to the sub-blocks of the current block. Alternatively, block vectors of the sub-blocks of the current block can be derived from the block vectors of neighboring blocks. The above derivation may be performed implicitly.
[0097] If surrounding blocks are encoded / decoded using a prediction vector-based prediction method, a prediction vector-based prediction method may be used for the current block. The prediction vector-based prediction method may be at least one of a motion vector-based prediction method, a block vector-based prediction method, and an affine mode.
[0098] For example, if all surrounding blocks do not use block vector prediction, the use of the block vector prediction method for the current block may be limited.
[0099] Alternatively, if all surrounding blocks do not use block vector prediction, the use of the motion vector prediction method for the current block may be limited.
[0100] Alternatively, if all surrounding blocks do not use motion vector prediction, the use of the block vector prediction method for the current block may be limited.
[0101] Alternatively, if all surrounding blocks do not use motion vector prediction, the use of the motion vector prediction method for the current block may be restricted.
[0102] This can be applied not only to the current block but also to the sub-blocks of the current block.
[0103] This can be applied not only to surrounding blocks but also to the sub-blocks of surrounding blocks.
[0104] The number of modes for prediction vector-based prediction can be determined by block information of the current block or surrounding blocks. The block information may include at least one of the width, height, area, or number of pixels within the block. Depending on the block information of the current block and surrounding blocks, the number of modes for prediction vector-based prediction may be limited.
[0105] For example, the prediction direction can be set to more directions than horizontal, vertical, and diagonal. Alternatively, it can be set to only some of the horizontal, vertical, and diagonal directions.
[0106] Alternatively, when the width and height of the current block differ, a specific prediction direction can be used more frequently. Specifically, the prediction direction based on the larger of the width and height can be used more frequently.
[0107] Alternatively, the number of prediction directions can be varied depending on the block area or aspect ratio.
[0108] Block vector-based prediction
[0109] The prediction vector for each sub-block of the current block can be derived based on the block vector or motion vector of the reference block. For example, the block vector for the sub-block of the current block can be derived based on the block vector of the reference block. The reference block may be a neighboring block adjacent to the current block. Alternatively, it may be a neighboring block not adjacent to the current block. Alternatively, it may be a block pointed to by the block vector of a neighboring block adjacent to the current block or a neighboring block not adjacent to the current block.
[0110] The derived prediction vector can be corrected using at least one of a template consisting of a reference block or pixels. For example, the block vector of the current block can be corrected based on the reference block. When correcting the block vector in this way, the block vector can be corrected within a correction range of any size. The arbitrary size representing the correction range may be in integer pixel units or subpixel units. Depending on the screen content compression information, the correction range may be limited to integer pixel units.
[0111] When a reference block is predicted by flipping, a prediction can be performed by performing a flipping on at least one of the prediction block, the template, or the current block. The flipping on the prediction block, the template, or the current block can be performed based on the flipping direction of the reference block. The flipping direction of the reference block can be signaled from the encoder or implicitly derived.
[0112] The predicted subblock of the current block can be generated based on the block vector of the reference block.
[0113] Alternatively, an inter-prediction mode or an intra-prediction mode of the current block can be derived based on the block vector of the reference block. Prediction can be performed based on the derived prediction mode of the current block.
[0114] Alternatively, template matching for the current block can be performed based on the block vector of the reference block.
[0115] Mean-based prediction
[0116] Mean-based prediction can be performed based on the prediction vector.
[0117] Average-based prediction may be based on at least one of average mode-based prediction, weighted average mode, motion vector-based mode, or block vector-based mode. The motion vector-based mode may be a mode having a prediction direction.
[0118] Which mode to use among average mode, weighted average mode, motion vector-based mode, or block vector-based mode can be determined based on a flag or index. The said flag or index can be signaled via a bitstream.
[0119] In the case of average mode or weighted average mode, whether to use it may be determined based on information that can induce a compression environment for the sequence. Possible compression environments for the sequence may include a Random Access compression environment, a Low Delay compression environment, or a Screen Content compression environment.
[0120] The averaging mode can derive a prediction vector for each sub-block by averaging the block vectors or motion vectors of surrounding blocks corresponding to each sub-block with equal weights. Alternatively, a prediction block can be generated by averaging the sample values of blocks indicated by the block vectors or motion vectors of surrounding blocks corresponding to each sub-block with equal weights.
[0121] The weighted average mode can generate a prediction vector for each sub-block by averaging the block vectors or motion vectors of surrounding blocks corresponding to each sub-block with a specific weight. Alternatively, it can generate a prediction block by averaging the sample values of blocks indicated by the block vectors or motion vectors of surrounding blocks corresponding to each sub-block with a specific weight. The weighted average mode can be derived based on at least one of the position of the sub-block, the distance between the sub-block and surrounding blocks, or the result of template matching.
[0122] Figure 4 shows the sub-blocks of the current block and surrounding blocks.
[0123] As an example of an average mode or weighted average mode, sub-block C1 of FIG. 4 can derive a predicted sample of sub-block C1 by weighted averaging the sample values of the blocks indicated by the block vectors of the upper surrounding block T1, the left surrounding block L1, and the upper-left surrounding block TL.
[0124] As another example, the predicted sample of sub-block C2 in Fig. 4 can be derived by weighted averaging the sample values of the blocks indicated by the block vectors of the upper surrounding block T2 and the left surrounding block L2.
[0125] As another example, the prediction vector of sub-block C2 in Fig. 4 can be derived based on a weighted average of the prediction vector of the upper surrounding block T2 and the prediction vector of the left surrounding block L2.
[0126] The prediction block generated using the above average mode or weighted average mode method may be the final prediction block.
[0127] Alternatively, a prediction block can be generated based on at least one of a general intra prediction mode, an inter prediction mode, or a prediction vector-based prediction mode. A final prediction block can be generated through a weighted average of the sample values of the prediction blocks generated by at least two of the prediction modes.
[0128] Reference image-based prediction
[0129] Prediction vector-based prediction can be performed even when the prediction vector points to a reference image. If the prediction vector points to a reference image, the prediction vector-based prediction can be a reference image-based prediction.
[0130] When performing reference image-based prediction, if the prediction vectors point to different reference images, the execution of reference image-based prediction can be restricted. That is, the encoder can restrict reference image-based prediction to be used only when the prediction vectors of surrounding blocks point to the same reference image.
[0131] When prediction vectors point to different reference images, reference image-based prediction can be performed. In this case, only a specific reference image may be used as the reference target. For example, prediction can be performed using prediction vectors that point to the same reference image. Specifically, prediction can be performed using a block vector-based mode, an average mode, or a weighted average mode based on prediction vectors pointing to the same reference image. In this case, a specific reference image can be selected based on the reference image with the smallest index within the reference picture list, or the reference image located closest to the current image.
[0132] Correction can be performed on the selected prediction vector.
[0133] The selected vectors can be corrected to point to a specific reference image. Here, the correction can be performed to point to each reference image or to point to only a specific reference image. The correction can be performed by scaling the vectors by considering the distance between the current image and each reference image or the specific reference image. Alternatively, the correction can be performed by adjusting the vectors through operations with an arbitrary vector.
[0134] Template matching can be performed during the process of performing prediction based on reference images.
[0135] Template matching can be performed on each reference image. Alternatively, template matching can be performed by scaling the vectors to use the same reference image when performing motion vector-based prediction.
[0136] If the prediction vector points in two or more directions or uses two or more reference picture lists, the prediction can be performed by utilizing all prediction vectors.
[0137] Alternatively, if the prediction vectors point to two or more directions or if two or more reference picture lists are used, predictions can be performed based on each set of prediction vectors. A set of prediction vectors may be a set in which prediction vectors are distinguished according to each direction, reference picture list, or reference image.
[0138] For example, there may be various candidate modes for a prediction vector-based prediction mode. In this case, the candidate modes may be derived based on at least one of a prediction direction, a reference picture list, or individual reference images. For example, if unidirectional prediction is required, a specific direction among multiple prediction directions may be selected to perform the prediction.
[0139] Vector interpolation-based prediction
[0140] When performing prediction based on prediction vectors, the final prediction vector can be derived by weighted averaging the vectors of surrounding blocks. Based on the final prediction vector, a prediction can be performed for the current block.
[0141] For example, a final predicted block can be derived by weighted averaging the vectors of the current subblock and referenceable neighbor blocks. In this case, whether a block is a referenceable neighbor block can be determined based on the angle of the prediction direction. The weights can be derived based on at least one of the angle of the prediction direction, the distance between the current subblock and neighbor blocks, or the result of template matching.
[0142] In the case where the vector is absent
[0143] Meanwhile, the motion vector of the reference block or surrounding blocks may not exist, or the block vector may not exist. Alternatively, parts of the surrounding blocks may not exist.
[0144] If a motion vector or block vector is absent, a vector of arbitrary size can be generated and assigned to a surrounding block. The assigned vector can be used as a prediction vector to perform prediction vector-based prediction for the current block.
[0145] Alternatively, the zero vector can be used.
[0146] Alternatively, prediction can be performed by using the vectors of other surrounding blocks where the motion vector or block vector exists.
[0147] Alternatively, a prediction execution vector can be derived by weighted averaging multiple vectors of other surrounding blocks where a motion vector or block vector exists, and a prediction can be performed based on the prediction execution vector.
[0148] Alternatively, the vector can be corrected through template matching, and a prediction can be performed based on the corrected vector. The vector subject to correction may be the zero vector or at least one of the arbitrary vectors.
[0149] If some of the surrounding blocks are missing, the reference target blocks can be generated by padding based on the motion vectors or block vectors of the surrounding blocks. Predictions for the current block can be performed based on the surrounding blocks generated through padding and the existing surrounding blocks.
[0150] Alternatively, a prediction vector can be derived by weighted averaging the block vectors of two or more surrounding blocks. Based on the prediction vector generated through the above derivation, a prediction for the current block can be performed.
[0151] Here, the case where part of the surrounding blocks does not exist may be when the current block is located on the boundary of a subpicture, slice, or tile.
[0152] Prediction vector correction
[0153] When performing a prediction based on at least one of the above prediction methods, a correction may be performed on the derived prediction vector or the final prediction vector.
[0154] The prediction vector can be corrected by a specific correction unit.
[0155] The correction unit may be one of a subblock unit, a unit larger than a subblock, a unit smaller than a subblock, or a pixel unit. Vector correction methods may include optical flow-based methods such as optical flow-based adjustment and frame optical flow refinement.
[0156] If the prediction vector points to both directions, it may be corrected by a specific correction unit by considering the difference between the pixels within the block pointed to by the two-way vector or the pixels pointed to by the prediction vector. The correction unit (and its magnitude) may be derived based on at least one of the difference between the pixels within the block pointed to by the two-way vector or the pixels pointed to by the two-way vector.
[0157] Correction of the prediction vector can be performed on a sub-block basis or on a pixel basis. Based on the corrected prediction vector, pixels within the prediction block (i.e., the sub-block of the current block) can be adjusted.
[0158] The resolution of the prediction vector can be corrected.
[0159] If the resolution of the prediction vector of surrounding blocks differs from the vector resolution required for prediction, the vector resolution can be corrected. Vector resolution correction can be performed through scaling.
[0160] Alternatively, if the resolution of the vector to be stored for encoding / decoding of a block decoded after the current block differs from the vector resolution required for prediction, the vector resolution can be corrected. Vector resolution correction can be performed through scaling.
[0161] When correcting vector resolution, the resolution of another vector can be corrected based on the resolution of one vector. The resolution serving as the reference for resolution correction may be at least one of the predicted vector resolution of a surrounding block, the vector resolution required for prediction, or the resolution of a vector that must be stored for encoding a block decoded after the current block.
[0162] For example, the vector resolution required for prediction can be determined by an adaptive motion or block vector resolution determination technique. In this case, the vector resolution of surrounding blocks can be corrected based on the said resolution. Alternatively, after prediction vector-based prediction is completed, the resolution of the vector that needs to be stored for encoding blocks decoded after the current block can be corrected.
[0163] Such methods for correcting prediction vectors can be applied to all of the above prediction methods.
[0164] Prediction mode extension
[0165] Meanwhile, the above prediction methods can be applied together with conventional prediction technologies.
[0166] If the current block is encoded based on the Geometric Partitioning Mode (GPM), a prediction vector-based prediction mode can be applied. In this case, prediction vector-based prediction can be performed in each partitioned region based on the GPM.
[0167] Alternatively, intra prediction or inter prediction can be performed in each segmented region based on GPM. If the motion vector of a subblock is a bidirectional prediction vector, reference image-based prediction can be performed by treating each vector as a separate vector. Alternatively, when performing intra prediction or IBC mode (Intra Block Copy Mode) in each segmented region, block vector-based prediction can be performed by using the block vector of the subblock as the prediction vector.
[0168] Other prediction techniques can also be applied when correcting prediction vectors. Template matching can be performed when correcting prediction vectors. Alternatively, residual vectors can be considered when correcting prediction vectors. In this case, correction can be performed for each prediction vector individually, or correction can be performed so that all or some of the prediction vectors have correction values of the same magnitude.
[0169] Overlapped Block Motion Compensation can be applied to prediction vector-based prediction methods. Prediction blocks can be generated based on the current subblock and the vectors of the surrounding left, right, top, and bottom subblocks. A new prediction block can be generated by weighted averaging each generated prediction block.
[0170] It is possible to induce whether to use Overlapped Block Motion Compensation for the left, right, top, and bottom subblocks surrounding the current subblock. The above induction may be performed implicitly. A prediction block can be generated using the vector of a subblock to which Overlapped Block Motion Compensation technology has been applied among the subblocks surrounding the current subblock. Overlapped Block Motion Compensation can be performed again on the prediction block generated in this way.
[0171] Alternatively, for subblocks surrounding the current subblock where Overlapped Block Motion Compensation is not used, a specific vector may be assigned and used. The specific vector may be a vector of a surrounding block or any arbitrary vector.
[0172] When performing a prediction vector-based prediction method, information regarding whether to use Overlapped Block Motion Compensation for encoding / decoding the next block can be stored.
[0173] By performing the aforementioned prediction vector-based prediction methods, a prediction block can be generated for each sub-block of the current block. To perform the prediction-based prediction method, one or more of the aforementioned prediction methods can be selected. Additionally, one or more prediction vector correction methods can be selected independently of the selection of the prediction method.
[0174] FIG. 5 is a block diagram illustrating a method of encoding an image based on an image encoding method performed in an image encoding device according to the present disclosure.
[0175] The image encoding method performed in the image encoding device according to the present disclosure may include the process of deriving a prediction vector for each sub-block of the current block (S500) and performing a prediction based on the prediction vector (S510). The method of deriving the prediction vector and the method of performing a prediction based on the prediction vector are as described in FIG. 3.
[0176] The various embodiments of the present disclosure are not intended to list all possible combinations but to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.
[0177] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, it may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.
[0178] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.
Claims
1. A step of deriving a prediction vector for each sub-block for the current block; and The method includes the step of performing a prediction for the current block based on the prediction vector, The above prediction vector includes at least one of a motion vector or a block vector, and An image decoding method in which the prediction for the current block is performed based on at least one of a block vector-based mode, an average mode, or a weighted average mode.
2. In Paragraph 1, Whether to perform the above prediction is determined based on at least one of the current block, block information of the surrounding blocks of the current block, or the prediction method of the surrounding blocks, and An image decoding method in which the above block information includes at least one of width, height, area, or aspect ratio.
3. In Paragraph 1, An image decoding method in which the number of predicted directions is determined based on at least one of the height or width of the current block.
4. In Paragraph 1, An image decoding method, wherein the step of deriving the above prediction vector includes the step of correcting the resolution of the above prediction vector.
5. In Paragraph 1, An image decoding method in which the above prediction vector is derived based on the block vector of a reference block.
6. In Paragraph 1, An image decoding method in which the above prediction vector is derived based on the weighted average of the prediction vectors of the surrounding blocks of the above current block.
7. In Paragraph 6, An image decoding method in which the above surrounding blocks are blocks coded based on at least one of prediction vector-based prediction, intra-prediction, or inter-prediction.
8. In Paragraph 1, The above prediction vector is derived based on a vector pointing to a first reference image, and An image decoding method in which the above prediction vector points only to the above first reference image.
9. In Paragraph 1, An image decoding method in which the above prediction vector is derived based on a vector pointing to a plurality of reference images.
10. In Paragraph 1, A video decoding method that performs the prediction for each partitioned region based on the Geometric Partitioning Mode (GPM) when the current block is encoded based on the above-mentioned Geometric Partitioning Mode (GPM).
11. A step of deriving prediction vectors for each sub-block for the current block; A step of correcting the above prediction vector; and The method includes the step of performing a prediction for the current block based on the corrected prediction vector, The above prediction vector includes at least one of a motion vector or a block vector, and The method of performing the above prediction is an image encoding method comprising at least one of a block vector-based mode, an average mode, or a weighted average mode.
12. A non-transient computer-readable storage medium for storing a bitstream generated by the image encoding method according to paragraph 11.
13. A method for transmitting a bitstream for transmitting a bitstream generated by the image encoding method according to paragraph 11.
