Video compression method and apparatus using reference vector-based intra mode prediction
By using reference vectors to derive and average candidate blocks for intra-frame prediction, the method addresses the inefficiencies in existing video compression technologies, enhancing the compression of high-resolution and stereoscopic video content.
Patent Information
- Application Number
- PCT/KR2026/096709
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
Existing video compression technologies struggle to efficiently handle high-resolution and stereoscopic video content, lacking effective methods for intra-frame prediction and block similarity determination.
The method and apparatus utilize reference vectors to derive reference and candidate blocks for intra-frame prediction, employing techniques like intra-block copy, template matching, and decoder-side mode derivation to determine similarity and generate a predicted block through weighted averaging.
This approach enhances the efficiency of video compression by improving intra-frame prediction accuracy and reducing redundancy, thereby optimizing the compression of high-resolution and stereoscopic video content.
Smart Images

Figure KR2026096709_12112026_PF_FP_ABST
Abstract
Description
Video compression method and device utilizing reference vector-based intra-frame mode prediction
[0001] The present invention relates to an image compression method and apparatus that performs in-frame prediction for the current block using a reference vector.
[0002] Market demand for high-resolution video is increasing, and consequently, there is a need for technology that can efficiently compress high-resolution video. In response to these market demands, the Moving Picture Expert Group (MPEG) of ISO / IEC and the Video Coding Expert Group (VCEG) of ITU-T jointly formed the Joint Collaborative Team on Video Coding (JCT-VC), completed the development of the High Efficiency Video Coding (HEVC) video compression standard in January 2013, and have been actively conducting research and development on next-generation compression standards.
[0003] Meanwhile, along with the increasing demand for high-resolution video, the demand for stereoscopic video content as a new video service is also rising. Discussions are underway regarding video compression technologies to effectively provide high-resolution and ultra-high-resolution stereoscopic video content.
[0004] The present disclosure aims to provide a method and apparatus for deriving a reference block and a candidate block for a current block.
[0005] The present disclosure aims to provide a method and apparatus for configuring a candidate block for a current block.
[0006] The present disclosure aims to provide a method and apparatus for determining similarity between candidate blocks.
[0007] The present disclosure aims to provide a method and apparatus for deriving a predicted block for a current block.
[0008] The image decoding method and apparatus according to the present disclosure can derive a reference block for a current block, derive a candidate block for the current block, and derive a prediction block based on the reference block and the candidate block.
[0009] In the image decoding method and apparatus according to the present disclosure, the reference block is derived based on a reference vector, and the reference vector may include at least one of a block vector or a motion vector.
[0010] In the image decoding method and apparatus according to the present disclosure, the reference vector is derived based on a candidate mode, and the candidate mode may include at least one of an intra-frame prediction mode, an inter-frame prediction mode, an intra-block copy mode, an intra-template matching mode, a decoder-side intra-mode derivation, or a template-based intra-mode derivation.
[0011] In the image decoding method and apparatus according to the present disclosure, the candidate block may be derived based on the reference vector or the candidate mode.
[0012] In the image decoding method and apparatus according to the present disclosure, the prediction block may be the block with the highest similarity to the reference block among the candidate blocks.
[0013] In the image decoding method and apparatus according to the present disclosure, the prediction block may be derived based on a weighted sum of a plurality of candidate blocks.
[0014] In the image decoding method and apparatus according to the present disclosure, if there are multiple candidate blocks with similar sample values among the candidate blocks, one of the multiple candidate blocks may be excluded from the candidate blocks.
[0015] In the image decoding method and apparatus according to the present disclosure, the similarity is determined based on a threshold value of the difference between the candidate blocks, and the threshold value can be derived based on the quantization parameters of the candidate blocks.
[0016] In the image decoding method and apparatus according to the present disclosure, the number of reference blocks or candidate blocks may be derived based on at least one of block information of the current block or the candidate mode.
[0017] In the image decoding method and apparatus according to the present disclosure, when the current block is a chrominance block, the prediction block may be derived by inheriting the prediction mode of the corresponding luminance block of the current block or all or part of the prediction block of the corresponding luminance block.
[0018] An image encoding method and apparatus according to the present disclosure can derive a reference block for a current block, derive a candidate block for the current block, and derive a prediction block based on the reference block and the candidate block.
[0019] A computer-readable recording medium according to the present disclosure can store a bitstream encoded by the image encoding method.
[0020] The bitstream transmission method and apparatus according to the present disclosure can transmit a bitstream generated by the image encoding method.
[0021] According to the present disclosure, reference blocks and candidate blocks can be derived for a current block.
[0022] According to the present disclosure, a predicted block of the current block can be derived based on the reference block and candidate block of the current block.
[0023] According to the present disclosure, a reference block can be derived based on the reference vector of the current block.
[0024] According to the present disclosure, a reference vector of the current block can be derived based on a candidate mode.
[0025] According to the present disclosure, similarity can be derived based on the difference value between a reference block and a candidate block.
[0026] According to the present disclosure, a final predicted block can be generated by performing a weighted sum based on the similarity between a reference block and a candidate block.
[0027] FIG. 1 is a block diagram showing an image encoding device according to the present disclosure.
[0028] FIG. 2 is a block diagram showing an image decoding device according to the present disclosure.
[0029] FIG. 3 is a block diagram illustrating a method for compressing an image based on reference vector-based in-frame mode prediction according to the present disclosure.
[0030] Figure 4 shows the reference vector in the in-frame mode prediction based on the reference vector.
[0031] Figure 5 shows various combinations of weighted averaging multiple candidate blocks.
[0032] Figure 6 illustrates the reference vector in cross-frame mode prediction based on the reference vector.
[0033] FIG. 7 is a block diagram illustrating a method for encoding an image based on reference vector-based in-frame mode prediction according to the present disclosure.
[0034] Embodiments of the present invention are described in detail with reference to the drawings attached to this specification so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.
[0035] Throughout this specification, when a part is described as being 'connected' to another part, this includes not only cases where they are directly connected, but also cases where they are electrically connected with other elements in between.
[0036] Furthermore, throughout this specification, when a part is described as 'comprising' a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.
[0037] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.
[0038] Additionally, in the embodiments relating to the device and method described herein, some components of the device or some steps of the method may be omitted. Also, the order of some components of the device or some steps of the method may be changed. Additionally, other components or other steps may be inserted into some components of the device or some steps of the method.
[0039] In addition, some components or steps of the first embodiment of the present invention may be added to or replace some components or steps of the second embodiment of the present invention.
[0040] Furthermore, the components shown in the embodiments of the present invention are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, for convenience of explanation, each component is described by listing it as a separate component, and at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function. Such integrated and separated embodiments of each component are also included within the scope of the present invention as long as they do not deviate from the essence of the invention.
[0041] In this specification, a block may be expressed in various ways such as a unit, region, unit, partition, etc., and a sample may be expressed in various ways such as a pixel, pel, pixel, etc. Pixel, pel, and pixel may have the same meaning.
[0042] Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. In describing the present invention, redundant descriptions of identical components are omitted.
[0043] FIG. 1 is a block diagram showing an image encoding device according to the present disclosure.
[0044] Referring to FIG. 1, the image encoding device (100) may include a picture splitting unit (110), a prediction unit (120, 125), a conversion unit (130), a quantization unit (135), a reordering unit (160), an entropy encoding unit (165), an inverse quantization unit (140), an inverse conversion unit (145), a filter unit (150), and a memory (155).
[0045] The picture splitting unit (110) can split the input picture into at least one processing unit. At this time, the processing unit may be a Prediction Unit (PU), a Transform Unit (TU), or a Coding Unit (CU). In the embodiments of the present disclosure below, the term "coding unit" may be used to mean a unit that performs coding or a unit that performs decoding.
[0046] A prediction unit may be divided into at least one shape, such as a square or rectangle, of the same size within a single encoding unit, or it may be divided such that one of the prediction units within a single encoding unit has a different shape and / or size from another prediction unit. When generating a prediction unit that performs intra-frame prediction based on an encoding unit, if it is not the minimum encoding unit, intra-frame prediction can be performed without dividing into multiple prediction units NxN.
[0047] The prediction unit (120, 125) may include an inter-frame prediction unit (120) that performs inter-frame prediction or inter-frame prediction, and an intra-frame prediction unit (125) that performs intra-frame prediction or intra-frame prediction. It may determine whether to use inter-frame prediction or perform intra-frame prediction for a prediction unit, and determine specific information according to each prediction method (e.g., intra-frame prediction mode, motion vector, reference picture, etc.). The residual value (residual block) between the generated prediction block and the original block may be input to the conversion unit (130). In addition, the prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value in the entropy encoding unit (165) and transmitted to the decoder.
[0048] The inter-frame prediction unit (120) may predict a prediction unit based on information of at least one picture among the previous picture or the subsequent picture of the current picture, and in some cases, may predict a prediction unit based on information of a partially encoded area within the current picture. The inter-frame prediction unit (120) may include a reference picture interpolation unit, a motion prediction unit, and a motion compensation unit.
[0049] In the reference picture interpolation unit, reference picture information is received from memory (155), and pixel information of integer pixels or less can be generated from the reference picture. In the case of luminance pixels, a DCT-based 8-tap interpolation filter with different filter coefficients can be used to generate pixel information of integer pixels or less in 1 / 4 pixel units. In the case of chrominance signals, a DCT-based 4-tap interpolation filter with different filter coefficients can be used to generate pixel information of integer pixels or less in 1 / 8 pixel units.
[0050] The motion prediction unit can perform motion prediction based on a reference picture interpolated by the reference picture interpolation unit. Various methods, such as FBMA (Full search-based Block Matching Algorithm), TSS (Three Step Search), and NTS (New Three-Step Search Algorithm), can be used to calculate motion vectors. Based on the interpolated pixels, the motion vector can have motion vector values in units of 1 / 2 or 1 / 4 pixels. The motion prediction unit can predict the current prediction unit by using different motion prediction methods. Various methods, such as Skip Mode, Merge Mode, AMVP Mode, Intra Block Copy Mode, Affine Mode, and Template Matching, can be used as motion prediction methods.
[0051] The in-frame prediction unit (125) can generate a prediction unit based on reference pixel information around the current block, which is pixel information within the current picture. If the surrounding block of the current prediction unit is a block that has performed inter-frame prediction, and the reference pixel is a pixel that has performed inter-frame prediction, the reference pixel included in the block that has performed inter-frame prediction can be replaced with the reference pixel information of the surrounding block that has performed in-frame prediction. That is, if the reference pixel is not available, the unavailable reference pixel information can be replaced with at least one of the available reference pixels.
[0052] Additionally, a residual block can be generated that includes residual value information, which is the difference between the prediction unit that performed the prediction based on the prediction unit generated in the prediction unit (120, 125) and the original block of the prediction unit. The generated residual block can be input to the conversion unit (130).
[0053] In the transformation unit (130), the residual block containing residual value information of the prediction unit generated through the original block and the prediction unit (120, 125) can be transformed using a transformation method such as DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), or KLT. Whether to apply DCT, DST, or KLT to transform the residual block can be determined based on the in-frame prediction mode information of the prediction unit used to generate the residual block.
[0054] The quantization unit (135) can quantize the values converted into the frequency domain in the conversion unit (130). The quantization coefficient may vary depending on the block or the importance of the image. The values produced by the quantization unit (135) may be provided to the inverse quantization unit (140) and the reordering unit (160).
[0055] The reordering unit (160) can perform reordering of coefficient values for quantized residual values.
[0056] The reordering unit (160) can convert two-dimensional block-shaped coefficients into one-dimensional vector forms through a coefficient scanning method. For example, the reordering unit (160) can convert the coefficients from DC to high-frequency ranges into one-dimensional vector forms by scanning using a Zig-Zag Scan method. Depending on the size of the conversion unit and the in-frame prediction mode, a vertical scan that scans two-dimensional block-shaped coefficients in the column direction or a horizontal scan that scans two-dimensional block-shaped coefficients in the row direction may be used instead of the Zig-Zag Scan. That is, depending on the size of the conversion unit and the in-frame prediction mode, it can be determined whether to use a Zig-Zag Scan, a vertical scan, or a horizontal scan.
[0057] The entropy encoding unit (165) can perform entropy encoding based on values calculated by the reordering unit (160). Entropy encoding can use various encoding methods, such as, for example, Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding). In this regard, the entropy encoding unit (165) can encode residual value coefficient information of the encoding unit from the reordering unit (160) and the prediction unit (120, 125). Additionally, according to the present disclosure, it is possible to signal and transmit information indicating that motion information is derived and used from the decoder side, and information regarding the technique used to derive motion information.
[0058] In the inverse quantization unit (140) and inverse transformation unit (145), the values quantized in the quantization unit (135) are inversely quantized, and the values transformed in the transformation unit (130) are inversely transformed. The residual value generated in the inverse quantization unit (140) and inverse transformation unit (145) can be combined with the predicted unit predicted through the motion estimation unit, motion compensation unit, and in-frame prediction unit included in the prediction unit (120, 125) to generate a reconstructed block.
[0059] The filter unit (150) may include at least one of a deblocking filter, an offset correction unit, and an Adaptive Loop Filter (ALF). The deblocking filter can remove block distortion caused by boundaries between blocks in the restored picture. The offset correction unit can correct the offset from the original image on a pixel-by-pixel basis for the image that has undergone deblocking. To perform offset correction for a specific picture, a method may be used in which pixels included in the image are divided into a certain number of regions, the region to be offset is determined, and the offset is applied to that region, or a method may be used in which the offset is applied by considering the edge information of each pixel. Adaptive Loop Filtering (ALF) may be performed based on a value obtained by comparing the filtered restored image with the original image. After dividing the pixels included in the image into a predetermined group, a filter to be applied to that group is determined, and filtering may be performed differentially for each group.
[0060] The memory (155) can store a restoration block or picture calculated through the filter unit (150), and the stored restoration block or picture can be provided to the prediction unit (120, 125) when performing inter-frame prediction.
[0061] FIG. 2 is a block diagram showing an image decoding device according to the present disclosure.
[0062] Referring to FIG. 2, the image decoding device (200) may include an entropy decoding unit (210), a reordering unit (215), an inverse quantization unit (220), an inverse transformation unit (225), a prediction unit (230, 235), a filter unit (240), and a memory (245).
[0063] When a video bitstream is input to a video encoding device, the input bitstream can be decoded by the reverse procedure of the video encoding device.
[0064] The entropy decoding unit (210) can perform entropy decoding in the opposite procedure to that which the entropy encoding unit of the image encoder performed. For example, various methods such as Exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) can be applied in correspondence with the method performed in the image encoder.
[0065] The entropy decoding unit (210) can decode information related to intra-frame prediction and inter-frame prediction performed by the encoder.
[0066] The reordering unit (215) can perform reordering based on the method of reordering the entropy-decoded bitstream in the encoding unit in the entropy decoding unit (210). The coefficients expressed in the form of a one-dimensional vector can be reordered by restoring them to the form of two-dimensional block coefficients.
[0067] The inverse quantization unit (220) can perform inverse quantization based on the coefficient values of the rearranged block and the quantization parameters provided by the encoder.
[0068] The inverse transform unit (225) can perform inverse transforms, i.e., inverse DCT, inverse DST, and inverse KLT, on the transforms, i.e., DCT, DST, and KLT, performed by the transform unit on the quantization result performed by the image encoder. The inverse transform can be performed based on the transmission unit determined by the image encoder. In the inverse transform unit (225) of the image decoder, a transformation technique (e.g., DCT, DST, KLT) can be selectively performed according to multiple pieces of information such as a prediction method, the size of the current block, and the prediction direction.
[0069] The prediction unit (230, 235) can generate a prediction block based on the prediction block generation information provided by the entropy decoding unit (210) and the previously decoded block or picture information provided by the memory (245).
[0070] As described above, when performing intra-frame prediction or intra-frame prediction identical to the operation in the video encoder, if the size of the prediction unit and the size of the transform unit are the same, intra-frame prediction for the prediction unit is performed based on the pixels to the left of the prediction unit, the pixels to the top left, and the pixels to the top; however, if the size of the prediction unit and the size of the transform unit are different when performing intra-frame prediction, intra-frame prediction can be performed using reference pixels based on the transform unit. Additionally, intra-frame prediction using NxN partitioning only for the minimum encoding unit may be used.
[0071] The prediction unit (230, 235) may include a prediction unit determination unit, an inter-frame prediction unit, and an intra-frame prediction unit. The prediction unit determination unit receives various information, such as prediction unit information input from the entropy decoding unit (210), prediction mode information of the intra-frame prediction method, and motion prediction related information of the inter-frame prediction method, distinguishes the prediction unit in the current encoding unit, and determines whether the prediction unit performs inter-frame prediction or intra-frame prediction. On the other hand, if the encoder (100) does not transmit motion prediction related information for the inter-frame prediction, but instead transmits information indicating that motion information is derived and used from the decoder side and information regarding the technique used for deriving motion information, the prediction unit determination unit determines whether the inter-frame prediction unit (230) performs prediction based on the information transmitted from the encoder (100).
[0072] The inter-frame prediction unit (230) can perform inter-frame prediction for the current prediction unit based on information included in at least one picture, either a previous picture or a subsequent picture, of the current picture containing the current prediction unit, using information required for inter-frame prediction of the current prediction unit provided by the video encoder. To perform inter-frame prediction, based on the encoding unit, it can determine whether the motion prediction method of the prediction unit included in the corresponding encoding unit is a Skip Mode, Merge Mode, AMVP Mode, Intra Block Copy Mode, Affine Mode, or Template Matching.
[0073] The in-frame prediction unit (235) can generate a prediction block based on pixel information within the current picture. If the prediction unit is a prediction unit that has performed in-frame prediction, it can perform in-frame prediction based on the in-frame prediction mode information of the prediction unit provided by the video encoder.
[0074] The intra-frame prediction unit (235) may include an Adaptive Intra Smoothing (AIS) filter, a reference pixel interpolation unit, and a DC filter. The AIS filter is a part that performs filtering on the reference pixels of the current block, and can determine whether to apply the filter based on the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block using the prediction mode of the prediction unit and AIS filter information provided by the image encoder. If the prediction mode of the current block is a mode that does not perform AIS filtering, the AIS filter may not be applied.
[0075] The reference pixel interpolation unit can generate a reference pixel of an integer value or less by interpolating the reference pixel when the prediction mode of the prediction unit is a prediction unit that performs intra-frame prediction based on the pixel value interpolated from the reference pixel. If the prediction mode of the current prediction unit is a prediction mode that generates a prediction block without interpolating the reference pixel, the reference pixel may not be interpolated. The DC filter can generate a prediction block through filtering when the prediction mode of the current block is DC mode.
[0076] The restored block or picture may be provided to a filter unit (240). The filter unit (240) may include a deblocking filter, an offset correction unit, and an ALF.
[0077] Information regarding whether a deblocking filter has been applied to the corresponding block or picture can be received from the video encoding device, and if a deblocking filter has been applied, information regarding whether a strong filter or a weak filter has been applied. The deblocking filter of the video decoder receives information related to the deblocking filter provided by the video encoder and can perform deblocking filtering on the corresponding block.
[0078] The offset correction unit can perform offset correction on the restored image based on the type of offset correction and offset value information applied to the image during encoding. ALF can be applied to the encoding unit based on information on whether ALF is applied and ALF coefficient information provided by the encoder. This ALF information can be provided included in a specific parameter set.
[0079] The memory (245) can store the restored picture or block so that it can be used as a reference picture or reference block, and can also provide the restored picture to the output unit.
[0080] In image coding, inter-frame prediction uses a technique that patches prediction blocks similar to the current block from the encoded / decoded reference picture to eliminate temporal redundancy. In contrast, intra-frame prediction generates prediction blocks based on reference samples.
[0081] FIG. 3 is a block diagram illustrating a method for compressing an image based on reference vector-based in-frame mode prediction according to the present disclosure.
[0082] According to FIG. 3, a method for compressing an image based on intra-frame mode prediction based on a reference vector according to the present disclosure may derive a reference block of the current block (S300), derive a candidate block of the current block (S310), and derive a prediction block based on the reference block and the candidate block (S320). Each step (S300, S310, S320) illustrated in FIG. 3 will be described below under the detailed description of each prediction method.
[0083] In-frame / inter-frame prediction modes may include a process of determining a reference sample, a process of determining a prediction mode, and a process of performing a prediction. On the other hand, template matching may include a process of finding the region most similar to the current block among some previously restored regions within the same picture, and restoring the current block based on this.
[0084] Example 1
[0085] Figure 4 shows the reference vector in the in-frame mode prediction based on the reference vector.
[0086] Referring to FIG. 4, in reference vector-based intra-frame mode prediction, the reference vector of the current block can be derived based on a predetermined intra-frame prediction mode. Here, the intra-frame prediction mode may include at least one of an Intra Block Copy (IBC) mode, an Intra Template Matching Prediction (Intra TMP) mode, a Decoder-side based Intra Mode Derivation (DIMD), or a Template-based Intra Mode Derivation (TIMD). The IntraTMP method is an intra-frame prediction technique in which an image decoder independently searches for and derives an optimal reference region using pre-recovered adjacent samples within the current frame to generate a prediction block of the current block. In this case, the template may be an adjacent sample adjacent to a pre-recovered region within the current frame. The template may be a pre-recovered adjacent sample.
[0087] Candidate blocks can be sorted by comparing the difference between the reference block and multiple candidate blocks generated in in-screen prediction mode for the current block.
[0088] The final prediction block can be derived by selecting the two candidate blocks with the smallest difference among the sorted candidate blocks and performing a weighted average. In this case, the weights can be derived based on the similarity between the reference block and each candidate block.
[0089] When generating a reference block in reference vector-based intra-frame mode prediction, the reference vector may be derived based on neighboring blocks or based on multiple intra-frame prediction modes. Alternatively, the block pointed to by the reference vector of the reference block may be used as a candidate block. The reference vector of the reference block may be derived based on the aforementioned intra-frame prediction mode or inter-prediction mode.
[0090] For example, a reference vector can be derived during the process of predicting the current block using an intra-block copy mode. The derived reference vector can be stored. The stored reference vector can be reused as a reference vector in a reference vector-based intra-frame mode prediction method. Alternatively, the reference vector can be corrected and used as a final reference vector. In this way, a reference block can be generated for the current block using the reference vector.
[0091] The number of reference blocks and candidate blocks can each be one or more.
[0092] If there is one reference block, one reference vector can be derived. In addition, one or more candidate blocks can be derived from the current block. Multiple final candidate blocks can be derived by weighted averaging one or more candidate blocks.
[0093] If there are multiple reference blocks, multiple reference vectors corresponding to each reference block can be derived. For example, if there are N reference blocks, N reference vectors corresponding to each reference block can be derived. N can be any natural number greater than or equal to 2. Additionally, one or more candidate blocks can be derived from the current block. In this case, multiple final candidate blocks can be derived by weighted averaging the one or more candidate blocks.
[0094] For both cases, a flag or index may be assigned to distinguish each of the multiple final candidate blocks. The said flag or index to distinguish each final candidate block may be signaled via a bitstream.
[0095] If multiple reference vectors or reference blocks exist, a flag or index may be assigned to distinguish each of the multiple reference vectors or reference blocks. The said flag or index to distinguish each reference vector or reference block may be signaled via a bitstream.
[0096] Alternatively, when applying a prediction method to the current block, a reference vector or a prediction mode may be applied depending on the type of prediction method. Candidate blocks can be derived based on the applied reference vector or prediction mode.
[0097] Alternatively, a candidate block can be derived based on the block pointed to by the reference vector of the reference block. The reference block may be determined based on the reference vector of the current block.
[0098] One of the candidate blocks derived by the above-described method may be used as the final prediction block. Alternatively, one block derived by weighted averaging multiple candidate blocks derived by the above-described method may be used as the final prediction block.
[0099] Multiple intra-frame prediction modes applied to the current block may include an intra-directional prediction mode, an intra-non-directional prediction mode, an intra-block copy (IBC) mode, an intra-template matching prediction (Intra TMP) mode, a decoder-side based intra-mode derivation (DIMD), or a template-based intra-frame prediction mode derivation (TIMD).
[0100] The above-described intra-frame prediction method applies in the same way even when multiple reference blocks and multiple candidate blocks are derived. For example, multiple reference blocks and candidate blocks may all be derived by the intra-frame prediction mode. Alternatively, the reference block may be derived through DIMD prediction, and multiple candidate blocks may be derived through the intra-frame prediction mode, DIMD, CIIP, IBC, etc. In this case, the reference block derived through DIMD prediction and the candidate blocks derived by the intra-frame prediction mode, DIMD, CIIP, and IBC can be compared to each other to select the blocks with the highest similarity. A final predicted block can be derived based on the weighted sum of the selected blocks.
[0101] Alternatively, multiple reference blocks may be generated based on one or more reference vectors, and one or more candidate blocks may be assigned to each reference block. For each reference block, the corresponding candidate blocks may be sorted in order of high similarity compared to the reference block. In this case, similarity may be the similarity between sample values of the blocks. One or more final prediction blocks may be derived by weighted averaging the candidate blocks in various combinations as shown in FIG. 5.
[0102] According to Fig. 5, each candidate block may be one of the above-derived multiple candidate blocks.
[0103] In addition, a weighted sum between candidate block 1 and candidate block 3 can be performed, or a weighted sum between candidate block 1 and candidate block 4 can be performed.
[0104] For example, candidate blocks can be sorted according to the difference value between the template region of the reference block and the template region of the candidate block, and a weighted average can be performed based on this. In this case, the reference block may be the block to which the candidate block is assigned. Methods for calculating the difference in sample values between the reference block and the candidate block may include the Sum of Absolute Transformed Difference (SATD), the Sum of Squared Difference (SSD), or the Mean Removed Sum of Difference (MRSAD). Candidate blocks can be sorted in order of smallest difference value.
[0105] Different weights can be assigned to each candidate block sorted in order of smallest difference value. The candidate block ultimately used for prediction can be derived based on a weighted sum of multiple candidate blocks. The weights assigned to each candidate block are given by the following Equation 1.
[0106]
[0107] Here, N is the total number of candidate blocks, and i may be the rank of the i-th sorted candidate block. Total_Weight may be the total sum of weights. Total_Weight may be a pre-defined value in the encoder / decoder. For example, the value of Total_Weight may be 1.
[0108] As another example, candidate blocks can be sorted based on the difference in directionality with respect to the block vector relative to the horizontal direction. For instance, if a candidate block exists in the same row as a reference block, it can be sorted as the first candidate block. If a candidate block exists in the same column as a reference block, it can be sorted as the last candidate block.
[0109] As another example, candidate blocks can be sorted based on the difference in directionality with respect to the block vector relative to the vertical direction. For instance, if a candidate block is in the same column as a reference block, it can be sorted to the first candidate block. If a candidate block is in the same row as a reference block, it can be sorted to the last candidate block.
[0110] Alternatively, multiple blocks can be generated based on one or more in-screen prediction modes, and the generated blocks can be weighted averaged to derive a block which can be used as a reference block.
[0111] Alternatively, multiple blocks can be generated based on one or more in-screen prediction modes, and the generated blocks can be weighted averaged to derive a block that can be used as a candidate block.
[0112] For example, a final reference block can be derived by weighted averaging reference blocks generated from two reference vectors. Alternatively, a reference block generated from reference vectors and a reference block generated by the DIMD method can be used as the final reference block by weighted averaging.
[0113] The similarity between a reference block and a candidate block can be derived from the difference in sample values between the reference block and the candidate block. Methods for calculating the difference in sample values between a reference block and a candidate block may include the Sum of Absolute Transformed Difference (SATD), the Sum of Squared Difference (SSD), or the Mean Removed Sum of Difference (MRSAD). For example, if Local Illumination Compensation (LIC) is applied to the reference block, the prediction block, or the template, the MRSAD calculation method may be used.
[0114] When deriving multiple prediction blocks based on multiple calculation methods, a flag or index can be assigned to each prediction block to distinguish them. The assigned flag or index can be signaled.
[0115] Alternatively, multiple candidate blocks can be derived based on multiple calculation methods. In this case, one of the multiple candidate blocks can be used as the final prediction block. Alternatively, a block derived by weighted averaging the multiple candidate blocks can be used as the final prediction block.
[0116] Example 2
[0117] The method for deriving the reference vector may include not only an intra-frame prediction mode but also an inter-frame prediction mode.
[0118] Figure 6 illustrates the reference vector in cross-frame mode prediction based on the reference vector.
[0119] Referring to Fig. 6, when a specific region of the restored image is selected as a reference block, the reference vector may be a motion vector.
[0120] Candidate blocks can be generated based on at least one of an intra-frame prediction mode, an inter-frame prediction mode, or an adaptive intra-frame to inter-frame prediction mode. The number of generated candidate blocks may be one or more.
[0121] Among the one or more generated candidate blocks, the single candidate block most similar to the reference block can be used as the final prediction block. Alternatively, the final prediction block can be derived by weighted averaging multiple candidate blocks.
[0122] When selecting a specific area among the restored regions within the same picture as a reference block, the reference vector can be a block vector.
[0123] The motion vector or block vector used to derive the reference block or candidate block can be corrected through template matching.
[0124] Candidate blocks can be generated based on at least one of an intra-frame prediction mode, an inter-frame prediction mode, or an adaptive intra-frame to inter-frame prediction mode. The number of generated candidate blocks may be one or more.
[0125] Among the one or more generated candidate blocks, the single candidate block most similar to the reference block can be used as the final prediction block. Alternatively, a block derived by weighted averaging multiple candidate blocks can be used as the final prediction block.
[0126] The block vector or motion vector used to derive the reference block or candidate block can be corrected through template matching.
[0127] The similarity between a reference block and a candidate block can be derived from the difference between the reference block and the candidate block. The method for deriving the difference between the reference block and the candidate block may be the same as the method described above.
[0128] For the prediction method of Example 1 or Example 2, a list for performing each prediction method can be configured. For example, a merged list of an in-frame prediction mode and an inter-frame prediction mode can be configured.
[0129] Alternatively, a mode for a specific prediction method or a list of vectors for a specific prediction method can be configured.
[0130] Alternatively, a list can be constructed for performing predictions.
[0131] Information regarding reference vector-based mode prediction may be added to each list. The information regarding reference vector-based mode prediction may include information regarding reference vector-based intra-frame mode prediction or information regarding reference vector-based cross-frame mode prediction.
[0132] If an existing list exists, information regarding reference vector-based mode prediction can be added to the existing list. If an existing list does not exist, information regarding reference vector-based mode prediction can be added to a separate list.
[0133] Information regarding reference vector-based mode prediction can be added to an existing list or a separate list. In this case, the information regarding reference vector-based mode prediction may include a weighted average mode in addition to each individual mode.
[0134] Each list may include modes available in a reference vector-based mode prediction method or pre-defined modes in other prediction methods. Each mode available in a reference vector-based mode prediction method or pre-defined mode in other prediction methods may be a candidate mode.
[0135] If each candidate mode is included in the same list, similar modes among them can be removed from the list.
[0136] For example, one of the two similar modes may be removed from the list through pruning based on at least one of the difference between blocks predicted according to each of the two similar candidate modes, the difference between the two modes, or the template matching result corresponding to the two modes. This may also apply when the number of similar candidate modes is three or more.
[0137] Whether to use a reference vector-based in-frame / cross-frame mode prediction method can be determined by block information of the current block or surrounding blocks. Block information may include the width or height of the block, area, or the number of pixels within the block.
[0138] For example, if the width is greater than A1 and the height is greater than B1, reference vector-based in-frame / cross-frame mode prediction may not be available. A1 and B1 can be any natural numbers.
[0139] Alternatively, if the width is less than A2 and the height is less than B2, reference vector-based in-frame / cross-frame mode prediction may not be available. A2 and B2 can be any natural numbers.
[0140] Alternatively, if the number of pixels within a block is less than C, reference vector-based in-frame / inter-frame mode prediction may not be available. C can be any positive number.
[0141] The mode of the candidate block or the number of candidate blocks may be determined by the block information of the current block or surrounding blocks. Alternatively, the mode of the candidate block or the number of candidate blocks may be determined by the prediction direction and type of the current block.
[0142] For example, if the width is less than A and the height is less than B, the number of candidate blocks may be N1, and otherwise, the number of candidate blocks may be N2. N1 and N2 can be any natural numbers.
[0143] Alternatively, if the area of a block is less than D, the number of candidate blocks may be M1, and if the area is greater than or equal to D, the number of candidate blocks may be M2. M1 and M2 can be any natural numbers.
[0144] Alternatively, some of the modes in the MPM (Most Probable Mode) list can be used for the current block.
[0145] Alternatively, you can use Primary MPM list mode or Planar mode for the current block.
[0146] Alternatively, for the current block, you can use some of the modes of the current block's left, top, top-left, bottom-left, and top-right blocks.
[0147] Alternatively, some of the modes in the MPM list or DIMD modes can be used for the current block.
[0148] Alternatively, some of the HoG (Histogram of Gradients)-based modes that utilize pixels from the surrounding blocks of the current block can be used.
[0149] Alternatively, at least one of MPM list, DIMD, IntraTMP, or TIMD may be used for the current block.
[0150] The number of pixels within the reference block and candidate blocks can be determined by the block information of the current block or surrounding blocks.
[0151] For example, the reference block and candidate block can be compared by downsampling, or by skipping N. N can be any natural number.
[0152] Alternatively, if the width is smaller than A and the height is smaller than B, the downsampling rate or the number of skipped pixels may be reduced. Otherwise, the downsampling rate or the number of skipped pixels may be increased.
[0153] Alternatively, if the number of pixels within a block is less than C, the downsampling rate or the number of skipped pixels may be reduced. Otherwise, the downsampling rate or the number of skipped pixels may be increased.
[0154] Alternatively, if the block area is smaller than D, the downsampling rate or the number of skipped pixels may be reduced. Otherwise, the downsampling rate or the number of skipped pixels may be increased.
[0155] Alternatively, when comparing a reference block and a candidate block based on at least one of block information of the current block or surrounding blocks, or the prediction direction and type of the current block, the position of the comparison sample may be set differently. For example, the position of the comparison sample may be set differently depending on whether it is in the horizontal or vertical direction, or the horizontal / vertical direction may be downsampled at different rates.
[0156] When deriving a candidate block based on at least one of the aforementioned embodiments, the candidate mode can be corrected based on the difference between the reference block and the candidate block. In this case, if there are multiple candidate blocks, the multiple candidate blocks can be sorted according to similarity compared with the reference block. For example, after deriving a candidate block based on DIMD, the DIMD mode can be corrected based on the difference between the candidate block and the reference block. The corrected DIMD mode can be used to derive the candidate block again later.
[0157] When adjusting the brightness between a reference block and a candidate block, the difference between the brightness value of the reference block and the brightness value of the candidate block can be induced by MRSAD.
[0158] Brightness can be adjusted using templates surrounding the reference block and candidate block, respectively. Alternatively, filter model parameters can be derived using templates surrounding the reference block and candidate block, respectively. The derivation of filter model parameters can be performed implicitly.
[0159] Some of the templates existing in the left, top, or top-left areas surrounding the current block can be adaptively selected. Alternatively, brightness adjustment can be performed on all templates existing in the left, top, or top-left areas surrounding the current block, and then the optimal template can be selected. Brightness adjustment can be performed based on the selected template.
[0160] Depending on whether surrounding blocks have adjusted their brightness or used a filter, it can be determined whether the current block's brightness is adjusted or a filter is used. Whether surrounding blocks' brightness is adjusted or a filter is used can be induced using the above template.
[0161] The final prediction method can be determined by considering both cases of brightness adjustment / unadjustment and filter usage / non-use. Information regarding whether brightness is adjusted or filters are used in the determined prediction method can be signaled in the form of a flag or index.
[0162] A reference vector-based intra-frame mode prediction method can be performed in conjunction with a combined inter-intra prediction (CIIP) method. In this case, intra-frame prediction can be performed using a block vector-based intra-frame mode prediction method (Example 1), and inter-frame prediction can be performed using a motion vector-based inter-frame mode prediction method (Example 2). Alternatively, both intra-frame prediction and inter-frame prediction can be performed using a block vector-based intra-frame mode prediction method.
[0163] When a reference vector-based intra-frame mode prediction method is applied together with a template-based intra-frame prediction method (TIMD), one or more modes may be applied, including an intra-frame prediction mode, a block vector-based prediction mode, and a mode based on a block vector-based mode prediction method. For example, a total of two modes may be used, consisting of one intra prediction mode and one mode based on a block vector-based mode prediction method. Alternatively, three modes determined by block vector-based mode prediction may be used. Or, a total of three modes may be used, consisting of two block vector-based prediction modes and one mode determined by block vector-based mode prediction.
[0164] The reference vector-based intra-frame mode prediction method can be used with the Geometric Partitioning Mode (GPM). When performing reference vector-based intra-frame prediction or reference vector-based inter-frame prediction in the Geometric Partitioning Mode, the difference in sample values between the corresponding reference block and candidate block for each partitioned region can be considered.
[0165] If the sample values of candidate blocks are similar to one another, similar candidate blocks can be removed from the candidate block list. In this case, two blocks can be defined as similar if the difference between their sample values does not exceed a threshold. The threshold can be derived based on the quantization parameters of the corresponding candidate blocks. Candidate blocks are considered similar if at least one of the following is smaller than an arbitrary threshold: the difference between the two candidate blocks, the difference between the candidate block modes, the difference between the prediction directions represented by the two candidate block modes, or the template matching result for the two candidate blocks. The removal of similar candidate blocks can be performed through pruning.
[0166] Reference vector-based intra-frame mode prediction or reference vector-based inter-frame mode prediction can be performed on the chrominance signal. In this case, the final candidate mode determined from the luminance signal can also be used for the chrominance signal. The chrominance signal may inherit the derivation method of the reference block or candidate block used in the luminance signal, the method for determining the optimal reference block or candidate block, etc.
[0167] Only some of the candidate modes and blocks used in the luminance signal can be used for the chrominance signal.
[0168] In the case of color difference blocks, a reference block and a candidate block can be generated based on the color difference signal and compared to generate a final prediction block.
[0169] Based on the compression information of the current block having a luminance signal, a color difference prediction block can be generated by using the final candidate mode determined from the luminance signal as the color difference signal prediction mode.
[0170] Alternatively, a reference block and a candidate block can be derived independently of the luminance signal, and a final prediction block can be derived based on the reference block and the candidate block.
[0171] FIG. 7 is a block diagram illustrating a method for encoding an image based on reference vector-based in-frame mode prediction according to the present disclosure.
[0172] Referring to FIG. 7, the method for encoding an image based on reference vector-based intra-frame mode prediction according to the present disclosure may derive a reference block of the current block (S700), derive a candidate block of the current block (S710), and derive a predicted block based on the reference block and the candidate block (S720). Each step of the method for encoding an image based on reference vector-based intra-frame mode prediction according to the present disclosure is as described in FIG. 3.
[0173] The various embodiments of the present disclosure are not intended to list all possible combinations but to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.
[0174] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, it may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.
[0175] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.
Claims
1. A step of deriving a reference block for the current block; A step of deriving a candidate block for the above current block; and An image decoding method comprising the step of deriving a prediction block based on the reference block and the candidate block.
2. In Paragraph 1, The above reference block is derived based on the reference vector of the above current block, and A video decoding method in which the above reference vector includes at least one of a block vector or a motion vector.
3. In Paragraph 2, The reference vector of the current block above is derived based on candidate modes, and A video decoding method comprising at least one of the above candidate modes, which include an intra-frame prediction mode, an inter-frame prediction mode, an intra-block copy mode, an intra-template matching mode, a decoder-side intra-mode derivation, or a template-based intra-mode derivation.
4. In Paragraph 3, An image decoding method in which the above candidate block is derived based on the reference vector of the above reference block.
5. In Paragraph 1, The above prediction block is the block with the highest similarity to the above reference block among the above candidate blocks, in an image decoding method.
6. In Paragraph 5, An image decoding method in which the above similarity is derived based on the difference value of sample values between the above reference block and the above candidate block.
7. In Paragraph 1, The above prediction block is an image decoding method derived based on a weighted sum of multiple candidate blocks.
8. In Paragraph 1, An image decoding method in which, if there are multiple candidate blocks with similar sample values among the above candidate blocks, one of the multiple candidate blocks is excluded from the candidate blocks.
9. An image decoding method according to claim 8, wherein the similarity is determined based on the difference value between the candidate blocks.
10. In Paragraph 4, An image decoding method in which the number of the reference block or the candidate block is derived based on at least one of the block information of the current block or the candidate mode.
11. In Paragraph 1, If the above current block is a color difference block, An image decoding method in which the above prediction block is derived by inheriting the prediction mode of the corresponding luminance block of the above current block or all or part of the prediction block of the above corresponding luminance block.
12. A step of deriving a reference block for the current block; A step of deriving a candidate block for the above current block; and An image encoding method comprising the step of deriving a prediction block based on the reference block and the candidate block.
13. A non-transient computer-readable storage medium for storing a bitstream generated by the image encoding method according to paragraph 13.
14. A method for transmitting a bitstream for transmitting a bitstream generated by the image encoding method according to paragraph 13.
