Method and apparatus for video coding
By adapting transformation and conversion kernels based on prediction modes and residual characteristics, the method enhances video coding efficiency, addressing high-quality video demands and reducing computational and storage costs.
Patent Information
- Application Number
- PCT/KR2026/096983
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-16
- Filing Date
- 2026-05-14
- Publication Date
- 2026-11-19
AI Technical Summary
Existing video coding technologies face challenges in achieving high coding efficiency due to increased demand for high-quality video, leading to higher computation, storage, and transmission costs, particularly in deriving directionality-related information and adapting transformation kernels for improved prediction accuracy.
The method involves deriving directionality-related information to adaptively determine transformation kernels and perform transformations based on prediction modes, using templates and templates to calculate costs for intra prediction modes, and optimizing conversion kernels for residual signals to enhance coding efficiency.
This approach improves video coding efficiency by reducing side information for intra prediction mode signaling, enhancing intra prediction performance, and optimizing conversion processes without increasing bit usage, thereby reducing computational and storage costs.
Smart Images

Figure KR2026096983_19112026_PF_FP_ABST
Abstract
Description
Video coding method and device
[0001] The present disclosure relates to an image / video coding method and an apparatus thereof.
[0002] In the context of video coding, the demand for high-quality video is increasing across various sectors of industry. This leads to an increase in the amount of information and throughput in areas such as computation, storage, and transmission, resulting in rising costs.
[0003] In relation to image / video coding technology, a method may be used to improve coding efficiency by deriving directionality-related information from reference samples and adaptively determining a transformation kernel based on this information to perform the transformation.
[0004] In recent high-efficiency video coding technologies, various intra-frame prediction modes are introduced to improve prediction accuracy, and methods for deriving prediction mode information from a decoder are being proposed to improve coding efficiency in the method of transmitting information about intra-frame prediction modes. Among these, TIMD is a method that quantifies the similarity between a restored template and a prediction template using a cost function, generates prediction blocks by selecting multiple prediction modes based on this, and derives a final prediction block by weighting the sum of the prediction blocks.
[0005] In addition, to improve the coding efficiency of the coding device, a method is proposed in which a first / second (inverse)transform kernel is determined based on the in-frame prediction mode of the current block, and a (inverse)transform is performed based on said kernel.
[0006] In addition, to improve the coding efficiency of the coding device, a method is proposed in which (inverse)conversion is performed based on an optimal conversion kernel for the residual signal of the current block.
[0007] According to one embodiment of the present disclosure, a method and apparatus for increasing video / image coding efficiency are provided.
[0008] According to one embodiment of the present disclosure, an intra-prediction-based video / image coding method and apparatus are provided.
[0009] According to one embodiment of the present disclosure, an image decoding method is provided by a decoding device. The method comprises the steps of: obtaining prediction-related information through a bitstream; deriving a prediction type for the current block based on the prediction-related information; deriving a template for the current block based on the prediction type; deriving candidate intra prediction modes based on the template for the current block; calculating a cost for each of the candidate intra prediction modes; deriving n intra prediction modes for the current block based on the cost of each of the candidate intra prediction modes; and generating a prediction block for the current block based on the n intra prediction modes for the current block. The template for the current block includes a left template and a top template, and the cost for the candidate intra prediction modes is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction modes.
[0010] According to one embodiment of the present disclosure, an image decoding method performed by a decoding device is provided. The method includes the step of deriving direction-related information based on a restored reference sample or a predicted sample, and the step of determining a transformation kernel based on the direction-related information.
[0011] According to one embodiment of the present disclosure, an image decoding method is provided by a decoding device. The method includes the step of deriving a characteristic of a residual signal (residual characteristic) based on a prediction mode or directionality of a current block or restored reference sample information, and the step of performing a conversion based on the residual characteristic.
[0012] According to one embodiment of the present disclosure, an image encoding method is provided by an encoding device. The method comprises the steps of determining a prediction type for a current block, deriving a template for the current block based on the prediction type, deriving candidate intra prediction modes based on the template for the current block, calculating a cost for each of the candidate intra prediction modes, deriving n intra prediction modes for the current block based on the cost of each of the candidate intra prediction modes, generating a prediction block for the current block based on the n intra prediction modes for the current block, generating prediction-related information based on the determined prediction type, and encoding image information including the prediction-related information. The template for the current block includes a left template and a top template, and the cost for the candidate intra prediction modes is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction modes.
[0013] According to one embodiment of the present disclosure, an image encoding method is provided by an encoding device. The method includes the step of deriving direction-related information based on a restored reference sample or a predicted sample, and the step of determining a transformation kernel based on the direction-related information.
[0014] According to one embodiment of the present disclosure, an image encoding method is provided by an encoding device. The method includes the step of deriving a characteristic of a residual signal (residual characteristic) based on a prediction mode or directionality of a current block or restored reference sample information, and the step of performing a transformation based on the residual characteristic.
[0015] According to one embodiment of the present disclosure, a decoding device for image decoding is provided. The decoding device comprises a memory and at least one processor connected to the memory, wherein the at least one processor is configured to perform the steps of: obtaining prediction-related information through a bitstream; deriving a prediction type for the current block based on the prediction-related information; deriving a template for the current block based on the prediction type; deriving candidate intra prediction modes based on the template for the current block; calculating a cost for each of the candidate intra prediction modes; deriving n intra prediction modes for the current block based on the cost of each of the candidate intra prediction modes; and generating a prediction block for the current block based on the n intra prediction modes for the current block, wherein the template for the current block includes a left template and a top template, and the cost for the candidate intra prediction mode is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction mode.
[0016] According to one embodiment of the present disclosure, a decoding device for image decoding is provided. The decoding device comprises a memory and at least one processor connected to the memory, wherein the at least one processor includes the step of deriving direction-related information based on a restored reference sample or a predicted sample, and the step of determining a transformation kernel based on the direction-related information.
[0017] According to one embodiment of the present disclosure, a decoding device for image decoding is provided. The decoding device comprises a memory and at least one processor connected to the memory, wherein the at least one processor includes the step of deriving a characteristic of a residual signal (residual characteristic) based on a prediction mode or directionality or restored reference sample information of a current block, and the step of performing a conversion based on the residual characteristic.
[0018] According to one embodiment of the present disclosure, an encoding device for video encoding is provided. The encoding device comprises a memory and at least one processor connected to the memory, wherein the at least one processor is configured to perform the steps of determining a prediction type for a current block, deriving a template for the current block based on the prediction type, deriving candidate intra prediction modes based on the template for the current block, calculating a cost for each of the candidate intra prediction modes, deriving n intra prediction modes for the current block based on the cost of each of the candidate intra prediction modes, generating a prediction block for the current block based on the n intra prediction modes for the current block, generating prediction-related information based on the determined prediction type, and encoding video information including the prediction-related information. The template for the current block includes a left template and a top template, and the cost for the candidate intra prediction modes is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction modes.
[0019] According to one embodiment of the present disclosure, an encoding device for video encoding is provided. The encoding device comprises a memory and at least one processor connected to the memory, wherein the at least one processor includes the step of deriving direction-related information based on a restored reference sample or a predicted sample, and the step of determining a transformation kernel based on the direction-related information.
[0020] According to one embodiment of the present disclosure, an encoding device for video encoding is provided. The encoding device comprises a memory and at least one processor connected to the memory, wherein the at least one processor includes the step of deriving a characteristic of a residual signal (residual characteristic) based on a prediction mode or directionality or restored reference sample information of a current block, and the step of performing a conversion based on the residual characteristic.
[0021] According to one embodiment of the present disclosure, a method for transmitting video / image data including a bitstream generated according to a video / image encoding method according to at least one of the embodiments of the present disclosure is provided.
[0022] According to one embodiment of the present disclosure, there is provided an apparatus for transmitting video / video data including a bitstream generated according to a video / video encoding method according to at least one of the embodiments of the present disclosure.
[0023] According to one embodiment of the present disclosure, a computer-readable storage medium may be provided that stores a program for performing a method according to at least one of the embodiments of the present disclosure.
[0024] According to one embodiment of the present disclosure, a computer-readable digital storage medium is provided that stores encoded video / image information generated according to a video / image encoding method according to at least one of the embodiments of the present disclosure.
[0025] According to one embodiment of the present disclosure, a computer-readable digital storage medium is provided that stores encoded information or encoded video / image information, which causes a video / image decoding method according to at least one of the embodiments of the present disclosure to be performed by a decoding device.
[0026] According to one embodiment of the present disclosure, overall video / image compression efficiency can be increased.
[0027] According to one embodiment of the present disclosure, the prediction performance for the current block can be improved.
[0028] According to one embodiment of the present disclosure, intra prediction of the current block can be performed based on a template.
[0029] According to one embodiment of the present disclosure, it is possible to improve intra prediction performance for the current block while reducing side information for intra prediction mode signaling.
[0030] According to one embodiment of the present disclosure, intra prediction modes are derived based on a template, and a prediction block for the current block can be generated through the fusion or weighted sum of prediction blocks using the same, and the intra prediction efficiency can be increased.
[0031] According to one embodiment of the present disclosure, the intra prediction performance can be improved by calculating the cost based on a template area that matches the intra prediction direction.
[0032] According to one embodiment of the present disclosure, the influence of an intra prediction mode with low contribution can be reduced, thereby improving intra prediction performance.
[0033] According to one embodiment of the present disclosure, the mismatch between the directionality of the intra prediction mode and the group classification can be reduced, thereby increasing the intra prediction efficiency.
[0034] According to one embodiment of the present disclosure, the fusion or weighted sum of unnecessary intra prediction blocks can be prevented, thereby increasing intra prediction efficiency.
[0035] According to one embodiment of the present disclosure, by determining a (inverse)conversion kernel optimized for the characteristics of the current conversion block, the efficiency of conversion and inverse conversion can be improved without increasing the number of bits.
[0036] According to one embodiment of the present disclosure, the efficiency of conversion and inverse conversion can be improved by determining a conversion method optimized for the residual characteristics of the current block.
[0037] FIG. 1 is a diagram schematically illustrating the configuration of a video / image encoding device to which embodiments of the present disclosure can be applied.
[0038] FIG. 2 is a diagram schematically illustrating the configuration of a video / image decoding device to which embodiments of the present disclosure can be applied.
[0039] Figure 3 illustrates an exemplary intra-prediction procedure.
[0040] Figure 4 illustrates an exemplary inter prediction procedure.
[0041] Figure 5 shows examples of intra-prediction-based video / image encoding methods.
[0042] Figure 6 shows examples of intra-prediction-based video / image decoding methods.
[0043] Figure 7 shows examples of directional intra prediction modes.
[0044] Figure 8 exemplarily illustrates the prediction directions associated with wide-angle intra prediction modes.
[0045] Figure 9 shows an example of the TIMD technique.
[0046] Figure 10 shows an example of a method for generating a TIMD prediction block.
[0047] Figure 11 shows an example of a method for determining the intra-prediction mode(s) to be used for prediction.
[0048] Figure 12 shows an example of generating prediction blocks by group.
[0049] Figure 13 shows an example of setting weights for prediction blocks by group when performing a pixel location-based weighted sum.
[0050] Figure 14 shows an example of performing a pixel-location-based weighted sum.
[0051] Figures 15a and 15b show examples of cost calculation based on the directionality of the prediction mode.
[0052] Figure 16 shows an example of cost calculation based on prediction mode directionality and template weights.
[0053] Figures 17a and 17b show examples of group classification correction.
[0054] Figure 18 shows an example of a TIMD technique with a different template configuration.
[0055] Figures 19a and 19b show examples of group classification with different template configurations.
[0056] Figures 20a and 20b show examples of group classification correction with different template configurations.
[0057] Figure 21 shows an example of a method for generating a TIMD prediction block with changed conditions.
[0058] Figure 22 illustrates first-order and second-order transformation techniques as examples.
[0059] FIG. 23 illustrates an exemplary residual processing procedure including a second transformation / inverse transformation.
[0060] Figure 24 illustrates an exemplary HoG (Histogram of Gradient) related to directionality.
[0061] FIG. 25 shows an example of a decoding procedure performed in a decoding device.
[0062] FIG. 26 exemplarily illustrates the procedure for determining the first / second inverse transform kernel in a coding device.
[0063] FIG. 27 illustrates an example of a method for determining a first / second inverse transform kernel for a non-regular intra block.
[0064] Figure 28 illustrates an exemplary method for deriving a HoG (Histogram of Gradient) related to directionality.
[0065] FIGS. 29 and 30 exemplarily show how a first-order inverse transform kernel is determined when the intensity of the directionality is evenly distributed.
[0066] FIGS. 31 and 32 exemplarily illustrate how a first-order inverse transform kernel is determined when the strength of one directionality is dominant.
[0067] FIG. 33 shows an example of a method for determining a first / second inverse transform kernel for a non-regular intra block.
[0068] FIG. 34 illustrates an example of a method for determining a first / second inverse transform kernel for a non-regular intra block.
[0069] FIG. 35 shows an example of an inverse conversion procedure performed in a coding device.
[0070] FIG. 36 illustrates the configuration of the inverse first-order transformation kernel determination unit (3520) as an example.
[0071] Figure 37 illustrates an exemplary process of performing an inverse transformation in the horizontal direction.
[0072] Figure 38 illustrates an exemplary region segmentation when the prediction mode of the current block is in-frame prediction.
[0073] FIG. 39 illustrates the configuration of a residual feature prediction unit (3540) in an exemplary manner.
[0074] Figure 40 illustrates the residual features in an exemplary manner.
[0075] Figure 41 exemplarily illustrates residual features when the prediction mode of the current block is in-frame prediction.
[0076] Figure 42 exemplarily illustrates residual features when the prediction mode of the current block is an inter-frame prediction mode.
[0077] Figure 43 exemplarily illustrates residual features when the prediction mode of the current block is a geometric partitioning prediction mode.
[0078] Figure 44 exemplarily illustrates the residual features when in-screen subdivision is applied to the current block.
[0079] Figure 45 exemplarily illustrates residual features when a sub-block transformation is applied to the current block.
[0080] FIG. 46 illustrates the configuration of the inverse first conversion unit (3550) as an example.
[0081] Figure 47 illustrates an inverse space transformation.
[0082] Figure 48 illustrates an exemplary method for deriving the degree of inverse space transformation (K).
[0083] FIG. 49 schematically illustrates a video / image encoding method according to the embodiment(s) of the present disclosure.
[0084] FIG. 50 schematically illustrates a video / image decoding method according to the embodiment(s) of the present disclosure.
[0085] As the present disclosure is subject to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments of the present disclosure to specific embodiments. Terms used in the present disclosure are used merely to describe specific embodiments and are not intended to limit the technical scope of the present disclosure. The singular forms used in the present disclosure are intended to include the plural forms unless the context clearly indicates otherwise. The term "and / or" used in the present disclosure includes any one or more combinations of the related listing items. The terms "comprising," "composing," and "holding" used in this specification specify the presence of the stated features, numbers, actions, elements, components, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, numbers, actions, elements, components, and / or combinations thereof. In this disclosure, the use of the term “may” in relation to examples or embodiments (e.g., what an example or embodiment may include or implement) means that there exists at least one example or embodiment in which such feature is included or implemented, but not all examples are limited thereto, and such feature or configuration may be omitted.
[0086] Meanwhile, each component in the drawings described in this disclosure is depicted independently for the convenience of explaining different characteristic functions and does not imply that each component is implemented in separate hardware or separate software. For example, two or more components may be combined to form a single component, or a single component may be divided into multiple components. Embodiments in which each component is integrated and / or separated are also included within the scope of this disclosure as long as they do not deviate from the essence of this disclosure.
[0087] In the present disclosure, "A or B" may mean "only A," "only B," or "both A and B." Alternatively, in the present disclosure, "A or B" may be interpreted as "A and / or B." For example, in the present disclosure, "A, B or C" may mean "only A," "only B," "only C," or "any combination of A, B and C."
[0088] A slash ( / ) or a comma used in the present disclosure may mean "and / or." For example, "A / B" may mean "A and / or B." Accordingly, "A / B" may mean "only A," "only B," or "both A and B." For example, "A, B, C" may mean "A, B or C."
[0089] In the present disclosure, "at least one of A and B" may mean "only A," "only B," or "both A and B." Additionally, in the present disclosure, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as synonymous with "at least one of A and B."
[0090] Additionally, in the present disclosure, "at least one of A, B and C" may mean "only A," "only B," "only C," or "any combination of A, B and C." Additionally, "at least one of A, B or C" or "at least one of A, B and / or C" may mean "at least one of A, B and C."
[0091] Additionally, parentheses used in the present disclosure may mean "for example." Specifically, when indicated as "prediction (intra-prediction)," "intra-prediction" may be proposed as an example of "prediction." In other words, the "prediction" of the present disclosure is not limited to "intra-prediction," and "intra-prediction" may be proposed as an example of "prediction." Furthermore, even when indicated as "prediction (i.e., intra-prediction)," "intra-prediction" may be proposed as an example of "prediction."
[0092] Technical features described individually within one drawing in this disclosure may be implemented individually or simultaneously.
[0093] The present disclosure relates to video / image coding. For example, the methods / exemplars described in the present disclosure may be applied to methods disclosed in the AV2 (AOMedia Video 2) standard. Additionally, the methods / exemplars disclosed in this disclosure may be applied to methods disclosed in the ECM (enhanced compression model) or H.267 standard, or next-generation video / image coding standards (e.g., H.268, H.269, etc.).
[0094] In the present disclosure, coding may include encoding and / or decoding. In the present disclosure, image coding may be used interchangeably with video coding.
[0095] In the present disclosure, "video" may refer to a set of images over time. "Frame" generally refers to a unit representing a single image at a specific time, and "slice" or "tile" is a unit that constitutes a part of a frame in coding. A slice or tile may include one or more superblocks. A single frame may be composed of one or more slices or tiles. A tile may represent a rectangular area of superblocks within a specific tile row or a specific tile column within a frame. Meanwhile, a single frame may be divided into two or more subframes. The term "frame" may be used interchangeably with "picture."
[0096] A pixel or pel can refer to the smallest unit that constitutes a single frame (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. Generally, a sample can represent a pixel or its value, and it may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component.
[0097] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a frame and information related to that region. A unit may include one luminance block and two chroma (e.g., cb, cr) blocks. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.
[0098] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the attached drawings. Hereinafter, the same reference numerals may be used for identical components in the drawings, and redundant descriptions of identical components may be omitted.
[0099] FIG. 1 is a diagram schematically illustrating the configuration of a video / image encoding device to which embodiments of the present disclosure may be applied. The term "encoding device" below may include an image encoding device and / or a video encoding device.
[0100] Referring to FIG. 1, the encoding device (100) may be configured to include an image partitioner (110), a predictor (120), a residual processor (130), an entropy encoder (140), an adder (150), a filter (160), and a memory (170). The predictor (120) may include an inter-predictor and an intra-predictor. The predictor (120) may include a neural network-based predictor. The neural network-based predictor may be included in the inter-predictor and / or intra-predictor. The residual processor (130) may include a transformer (132), a quantizer (133), a dequantizer (134), and an inverse transformer (135). The residual processing unit (130) may further include a subtractor (131). The image segmentation unit may be called a picture segmentation unit. The addition unit (150) may be called a reconstructor or a reconstructed block generator. The above-described image segmentation unit (110), prediction unit (120), residual processing unit (130), entropy encoding unit (140), addition unit (150), and filtering unit (160) may be configured by one or more hardware components (e.g., an encoder chipset or processor) according to the embodiment. Additionally, the memory (170) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (170) as an internal / external component.
[0101] The image segmentation unit (110) can divide an input image (or picture, frame) input to an encoding device (100) into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (Quad-tree binary-tree ternary-tree) structure. For example, one coding unit may be divided into multiple coding units of a deeper depth based on a quad-tree structure, a binary-tree structure, and / or a ternary structure. In this case, for example, the quad-tree structure may be applied first and the binary-tree structure and / or ternary structure may be applied later. Or the binary-tree structure may be applied first. A coding procedure according to the present disclosure may be performed based on the final coding unit that is no longer divided. In this case, based on coding efficiency according to image characteristics, the maximum coding unit may be used directly as the final coding unit, or, if necessary, the coding unit may be recursively divided into lower-depth coding units so that a coding unit of the optimal size is used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may each be divided or partitioned from the final coding unit described above.The above prediction unit may be a unit for sample prediction, and the above transformation unit may be a unit for deriving transformation coefficients and / or a unit for deriving a residual signal from transformation coefficients.
[0102] The term "unit" may be used interchangeably with terms such as "block" or "area" depending on the context. In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chroma component. A sample may be used to refer to a single picture (or image) as a term corresponding to a pixel or pel.
[0103] The encoding device (100) can generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from a prediction unit from an input image signal (original block, original sample array), and the generated residual signal is transmitted to a conversion unit (132). In this case, as illustrated, the unit that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) within the encoding device (100) may be called a subtraction unit (131). The prediction unit can perform a prediction on a block to be processed (hereinafter referred to as the current block) and generate a predicted block containing prediction samples for the current block. The prediction unit can determine whether intra prediction is applied or inter prediction is applied at the current block or CU level. The prediction unit can generate various information regarding prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit it to the entropy encoding unit (140). The information regarding prediction can be encoded in the entropy encoding unit (140) and output in the form of a bitstream.
[0104] The intra prediction unit can predict the current block by referencing samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a Planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes, depending on the degree of fineness of the prediction direction. However, this is merely an example, and depending on the settings, a number of more or fewer directional prediction modes may be used. The intra prediction unit may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.
[0105] The inter-prediction unit can derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter-prediction mode, motion information can be predicted in block, sub-block, or sample units based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter-prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different. The above temporal surrounding blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc., and the reference picture containing the above temporal surrounding blocks may be referred to as a collocated picture (colPic). For example, the inter prediction unit may construct a list of motion information candidates based on surrounding blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction may be performed based on various prediction modes; for example, in the case of skip mode and merge mode, the inter prediction unit may use the motion information of surrounding blocks as the motion information of the current block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted.In the motion vector prediction (MVP) mode, the motion vector of surrounding blocks is used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.
[0106] The prediction unit (120) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for the prediction of a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit may be based on an intra block copy (IBC) prediction mode or a palette mode for the prediction of a block. The IBC prediction mode or palette mode may be used, for example, for screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block based on a block vector within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this disclosure.
[0107] The prediction signal generated through the prediction unit (120) can be used to generate a restoration signal or to generate a residual signal. The transformation unit (132) can generate transform coefficients by applying a transformation technique to the residual signal. For example, the transformation technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), or a Conditionally Non-linear Transform (CNT).
[0108] The quantization unit (133) quantizes the transformation coefficients and transmits them to the entropy encoding unit (140), and the entropy encoding unit (140) can encode the quantized signal (information regarding the quantized transformation coefficients) and output it as a bitstream. The information regarding the quantized transformation coefficients may be called residual information. The quantization unit (133) can rearrange the block-shaped quantized transformation coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate information regarding the quantized transformation coefficients based on the one-dimensional vector-shaped quantized transformation coefficients. The entropy encoding unit (140) can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding), etc. The entropy encoding unit (140) may encode information necessary for video / image restoration (e.g., values of syntax elements) in addition to quantized transform coefficients, either together or separately. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of a network abstraction layer (NAL) unit. The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. In this disclosure, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded through the encoding procedure described above and included in the bitstream.The above bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmission unit (not shown) that transmits the signal output from the entropy encoding unit (140) and / or a storage unit (not shown) that stores it may be configured as internal / external elements of the encoding device (100), or the transmission unit may be included in the entropy encoding unit (140).
[0109] Quantized transformation coefficients output from the quantization unit (133) can be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) can be restored by applying inverse quantization and inverse transformation to the quantized transformation coefficients through the inverse quantization unit (134) and the inverse transformation unit (135). An adder (150) can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the prediction unit. In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block. The adder (150) may be called a reconstructed unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra-prediction of the next block to be processed within the current picture, and can also be used for inter-prediction of the next picture after undergoing filtering as described below.
[0110] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or restoration process.
[0111] The filtering unit (160) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (160) can generate a modified restored picture by applying various filtering methods to the restored picture, and can store the modified restored picture in memory (170), specifically in the DPB of memory (170). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit (160) can generate information regarding filtering and transmit it to the entropy encoding unit (140). The information regarding filtering can be encoded in the entropy encoding unit (140) and output in the form of a bitstream.
[0112] The modified restored picture transmitted to memory (170) can be used as a reference picture in the inter-prediction unit. Through this, the encoding device (100) can avoid prediction mismatches between the encoding device (100) and the decoding device when inter-prediction is applied, and can also improve encoding efficiency.
[0113] The memory (170) DPB can store the modified restored picture to be used as a reference picture in the inter-prediction unit. The memory (170) can store motion information of blocks from which motion information is derived (or encoded) within the current picture and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (170) can store restoration samples of the blocks restored within the current picture and transmit them to the intra-prediction unit.
[0114] Encoded video / image information or data output in bitstream form may be transmitted to a decoding device via a network in the form of a file or streaming. Encoded video / image information or data output in bitstream form may also be transmitted to a decoding device via a digital storage medium or a streaming server. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A streaming server may transmit encoded video / image information or data output in bitstream form via a wired or wireless communication network. A transmission unit / transmission device may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. As video becomes high-resolution and high-quality, the size of the original video / image data increases, and by acquiring and storing / transmitting a bitstream (or data including a bitstream) generated through the efficient encoding method according to the present disclosure, storage / transmission efficiency can be increased and low-latency / real-time transmission can be supported.
[0115] The streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream. The streaming server transmits multimedia data to a user device based on a user request through a web server, and the web server acts as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server forwards this request to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server plays the role of controlling commands and responses between each device within the content streaming system.
[0116] The streaming server can receive content from a media storage and / or encoding device. For example, when receiving content from the encoding device, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store the bitstream for a certain period of time.
[0117] FIG. 2 is a diagram schematically illustrating the configuration of a video / image decoding device to which embodiments of the present disclosure may be applied. The term "decoding device" below may include an image decoding device and / or a video decoding device.
[0118] Referring to FIG. 2, the decoding device (200) may be configured to include an entropy decoder (210), a residual processor (220), a predictor (230), an adder (240), a filter (250), and a memory (260). The predictor (230) may include an inter-predictor and an intra-predictor. The predictor (230) may include a neural network-based predictor. The neural network-based predictor may be included in the inter-predictor and / or intra-predictor. The residual processor (220) may include a dequantizer (221) and an inverse transformer (221). The aforementioned entropy decoding unit (210), residual processing unit (220), prediction unit (230), addition unit (240), and filtering unit (250) may be configured by a single hardware component (e.g., a decoder chipset or a processor) according to an embodiment. Additionally, the memory (260) may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware component may further include the memory (260) as an internal / external component.
[0119] When a bitstream containing video / image information is input, the decoding device (200) can restore the image in correspondence with the process in which the video / image information is processed by the encoding device. For example, the decoding device (200) can derive units / blocks based on block division information obtained from the bitstream. The decoding device (200) can perform decoding using a processing unit applied by the encoding device. Accordingly, the processing unit for decoding may be, for example, a coding unit, and the coding unit may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a binary tree structure. One or more transformation units may be derived from the coding unit. And, the restored image signal decoded and output through the decoding device (200) can be played back through a playback device.
[0120] A decoding device (200) can receive a signal output from an encoding device in the form of a bitstream, and the received signal can be decoded through an entropy decoding unit (210). For example, the entropy decoding unit (210) can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Additionally, the video / image information may further include general constraint information. The decoding device can decode the picture based further on the information regarding the parameter sets and / or the general constraint information. The signaling / receiving information and / or syntax elements described below in this disclosure may be decoded through the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit (210) can decode information within a bitstream based on coding methods such as exponential chord coding, CAVLC, or CABAC, and output values of syntax elements required for image restoration and quantized values of transformation coefficients regarding residuals. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in a bitstream, determine a context model using information of the syntax element to be decoded and decoding information of surrounding and decoding target blocks or information of symbols / bins decoded in the previous step, predict the probability of occurrence of the bin according to the determined context model, and perform arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element.At this time, the CABAC entropy decoding method can update the context model using the decoded symbol / bin information for the context model of the next symbol / bin after determining the context model. Among the information decoded in the entropy decoding unit (210), information regarding prediction is provided to the prediction unit (230), and residual values for which entropy decoding was performed in the entropy decoding unit (210), i.e., quantized transformation coefficients and related parameter information, can be input to the residual processing unit (220). The residual processing unit (220) can derive residual signals (residual blocks, residual samples, residual sample array). Additionally, among the information decoded in the entropy decoding unit (210), information regarding filtering can be provided to the filtering unit (250). Meanwhile, a receiving unit (not shown) that receives a signal output from an encoding device may be further configured as an internal / external element of the decoding device (200), or the receiving unit may be a component of the entropy decoding unit (210). Meanwhile, the decoding device according to the present disclosure may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoding unit (210), and the sample decoder may include at least one of the inverse quantization unit (221), inverse transform unit (222), adder (240), filtering unit (250), memory (260), and prediction unit (230).
[0121] In the inverse quantization unit (221), the quantized transform coefficients can be inversely quantized to output transform coefficients. The inverse quantization unit (221) can rearrange the quantized transform coefficients into a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit (221) can perform inverse quantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain transform coefficients.
[0122] In the inverse conversion unit (222), the conversion coefficients are inversely converted to obtain a residual signal (residual block, residual sample array).
[0123] The prediction unit performs a prediction for the current block and can generate a predicted block containing prediction samples for the current block. Based on information regarding the prediction output from the entropy decoding unit (210), the prediction unit can determine whether an intra prediction or an inter prediction is applied to the current block and can determine a specific intra / inter prediction mode.
[0124] The prediction unit (230) can generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for a single block, and may also apply intra prediction and inter prediction simultaneously. This may be called combined inter and intra prediction (CIIP). Additionally, the prediction unit (230) may be based on an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used, for example, for screen content coding (SCC). IBC basically performs prediction within the current picture, but it may be performed similarly to inter prediction in that it derives a reference block based on a block vector within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this disclosure.
[0125] The intra prediction unit can predict the current block by referring to samples within the current picture. The referenced samples may be located near the current block or away from it, depending on the prediction mode. In intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit may also determine the prediction mode applied to the current block by using the prediction mode applied to the surrounding blocks.
[0126] The inter prediction unit can derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. At this time, to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in block, sub-block, or sample units based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit may construct a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference picture index of the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the mode of inter prediction for the current block.
[0127] The adder (240) can generate a restoration signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the prediction signal (predicted block, predicted sample array) output from the prediction unit (230). In cases where there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the restoration block.
[0128] The addition unit (240) may be called a restoration unit or a restoration block generation unit. The generated restoration signal may be used for intra-predicting the next block to be processed within the current picture, may be output after filtering as described below, or may be used for inter-predicting the next picture.
[0129] Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.
[0130] The filtering unit (250) can improve subjective / objective image quality by applying filtering to the restored signal. For example, the filtering unit (250) can generate a modified restored picture by applying various filtering methods to the restored picture, and can transmit the modified restored picture to memory (260), specifically to the DPB of memory (260). The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.
[0131] The (modified) restored picture stored in the DPB of the memory (260) can be used as a reference picture in the inter-prediction unit. The memory (260) can store motion information of blocks from which motion information within the current picture has been derived (or decoded) and / or motion information of blocks within the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit to be used as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. The memory (260) can store restoration samples of the blocks restored within the current picture and transmit them to the intra-prediction unit.
[0132] In this specification, the operations described for the filtering unit (160), prediction unit (120), inverse quantization unit (134), inverse transform unit (135), etc. of the encoding device (100) may be applied in the same or correspondingly to the filtering unit (250), prediction unit (230), inverse quantization unit (221), inverse transform unit (222), etc. of the decoding device (200). Additionally, the operations described for the transform unit (132), quantization unit (133), etc. of the encoding device (100) may be performed in a corresponding manner to the inverse quantization unit (221), inverse transform unit (222), etc. of the decoding device (200).
[0133] As described above, prediction is performed to increase compression efficiency during video coding. Through this, a predicted block containing predicted samples for the current block, which is the block to be coded, can be generated. Here, the predicted block includes predicted samples in the spatial domain (or pixel domain). The predicted block is derived identically by both the encoding device and the decoding device, and the encoding device can increase video coding efficiency by signaling information regarding the residual between the original block and the predicted block (residual information) to the decoding device, rather than the original sample value of the original block itself. The decoding device derives a residual block containing residual samples based on the residual information, can generate a restored block containing restored samples by combining the residual block and the predicted block, and can generate a restored picture containing the restored blocks.
[0134] The above residual information can be generated through transformation and quantization procedures. For example, an encoding device may derive a residual block between the original block and the predicted block, perform a transformation procedure on residual samples (residual sample array) included in the residual block to derive transformation coefficients, perform a quantization procedure on the transformation coefficients to derive quantized transformation coefficients, and signal the related residual information to a decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, transformation technique, transformation kernel, and quantization parameters of the quantized transformation coefficients. The decoding device may perform an inverse quantization / inverse transformation procedure based on the residual information and derive residual samples (or residual blocks). The decoding device may generate a reconstructed picture based on the predicted block and the residual block. The encoding device can also derive a residual block by inversely quantizing / inversely transforming the quantized transform coefficients for reference to inter-predicting of the picture, and generate a restored picture based thereon.
[0135] In the present disclosure, at least one of quantization / inverse quantization and / or transformation / inverse transformation may be omitted. If the quantization / inverse quantization is omitted, the quantized transformation coefficient may be referred to as a transformation coefficient. If the transformation / inverse transformation is omitted, the transformation coefficient may be referred to as a coefficient or residual coefficient, or may still be referred to as a transformation coefficient for the sake of consistency of expression.
[0136] Additionally, in this disclosure, quantized transform coefficients and transform coefficients may be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, residual information may include information regarding transform coefficient(s), and information regarding said transform coefficient(s) may be signaled through residual coding syntax. Transform coefficients may be derived based on said residual information (or information regarding said transform coefficient(s), and scaled transform coefficients may be derived through inverse transform (scaling) of said transform coefficients. Residual samples may be derived based on inverse transform (transform) of said scaled transform coefficients. This may be similarly applied / expressed in other parts of this disclosure.
[0137] Intra prediction may represent a prediction that generates prediction samples for the current block based on reference samples within the picture to which the current block belongs (hereinafter, the current picture). When intra prediction is applied to the current block, surrounding reference samples to be used for the intra prediction of the current block may be derived. The surrounding reference samples of the current block may include H+W samples located to the left of the current block of size W / XH, W+H samples located to the top of the current block, and at least one sample adjacent to the top-left of the current block. Alternatively, the surrounding reference samples of the current block may include multiple rows of upper surrounding samples and multiple columns of left surrounding samples.
[0138] Some of the surrounding reference samples in the current block may not yet be decoded or may not be available. In this case, the decoder can construct the surrounding reference samples to be used for prediction by padding or substituting the unavailable samples with the available samples.
[0139] When surrounding reference samples are derived, prediction samples of the current block can be derived based on the surrounding reference samples and intra prediction mode / type information. Here, the intra prediction mode may represent one of non-directional prediction modes and directional prediction modes that indicate spatial correlations for intra prediction. Here, the directional prediction mode may be called an angular prediction mode, and the non-directional prediction mode may be called a non-angular prediction mode. The intra prediction type may represent various prediction types for performing intra prediction. Intra prediction types may include, for example, MRL (multi-reference line), ISP (intra sub-partitions), PDPC (Position dependent intra prediction), MIP (matrix weighted intra prediction or matrix based intra prediction), CCLM (cross-component linear model), MMLM (multi-model linear model), DIMD (Decoder side intra mode derivation), fusion of chroma intra prediction modes, intra template matching, TIMD (fusion for template-based intra mode derivation), intra prediction fusion, CCCM (cross-component convolutional model), CCP (cross-component prediction), SGPM (spatial geometric partitioning mode), etc. Depending on the case, intra prediction modes and / or intra prediction types may be used to perform intra prediction.
[0140] Specifically, the intra-prediction procedure may include an intra-prediction mode / type determination step, a reference sample derivation step, and an intra-prediction mode / type-based prediction sample derivation step. Additionally, a post-filtering step for the derived prediction samples may be performed as needed.
[0141] Figure 3 illustrates an exemplary intra-prediction procedure.
[0142] Referring to FIG. 3, as described above, the intra prediction procedure may include an intra prediction mode / type determination step, a reference sample derivation step, and an intra prediction execution (prediction sample generation) step. The intra prediction procedure may be performed in an encoding device and a decoding device as described above.
[0143] The coding device derives an intra-prediction mode / type for the current block (S300). The coding device may include an encoding device and / or a decoding device as described above.
[0144] The encoding device can determine an intra prediction mode / type applied to the current block among the various intra prediction modes / types described in the present disclosure and can generate prediction-related information. The prediction-related information may include intra prediction mode information indicating an intra prediction mode applied to the current block and / or intra prediction type information indicating an intra prediction type applied to the current block. The decoding device can determine an intra prediction mode / type applied to the current block based on the prediction-related information.
[0145] For example, when intra prediction is applied, the intra prediction mode applied to the current block can be determined using the intra prediction mode of the surrounding blocks. For example, the coding device may select one of the MPM candidates within the list of most probable modes (MPM) derived based on the intra prediction modes of the surrounding blocks of the current block (e.g., left and / or upper surrounding blocks) and / or additional candidate modes based on received index information, or select one of the remaining intra prediction modes not included in the MPM candidates based on MPM retainer information (remaining intra prediction mode information). The MPM list may be configured to include or not include the planner mode as a candidate.
[0146] A list of most probable modes (MPM) for the current block of the coding device can be constructed. The above MPM list may also be referred to as an MPM candidate list. Here, MPM may refer to a mode used to improve coding efficiency by considering the similarity between the current block and surrounding blocks during intra-predictive mode coding.
[0147] The encoding device can perform predictions based on various intra prediction modes and determine the optimal intra prediction mode based on rate-distortion optimization (RDO) derived therefrom. In this case, the encoding device may determine the optimal intra prediction mode using MPM candidates configured in the MPM list, or it may determine the optimal intra prediction mode using the remaining intra prediction modes other than the MPM candidates configured in the MPM list. Specifically, for example, if the intra prediction type of the current block is a specific type (e.g., DIMD, TIMD, MRL, or ISP) rather than a normal intra prediction type, the encoding device may determine the optimal intra prediction mode by considering only specific candidates as intra prediction mode candidates for the current block. The specific candidates may include some or all of the MPM candidates. For example, in this case, the intra prediction mode for the current block may be determined only from among the MPM candidates, and in this case, the MPM flag may not be encoded / signaled. In this case, the decoding device can assume that the MPM flag is 1 without separately receiving the MPM flag signal.
[0148] Meanwhile, generally, if the intra prediction mode of the current block is not a planner mode and is one of the MPM candidates in the MPM list, the encoding device generates an mpm index (mpm idx) pointing to one of the MPM candidates. If the intra prediction mode of the current block is not even in the MPM list, it generates MPM retainer information (remaining intra prediction mode information) pointing to a mode that is the same as the intra prediction mode of the current block among the remaining intra prediction modes not included in the MPM list (and planner mode). The MPM retainer information may include, for example, an intra_luma_mpm_remainder syntax element.
[0149] The decoding device obtains intra prediction mode information from the bitstream. As described above, the intra prediction mode information may include at least one of an MPM flag, an MPM index, and MPM retainer information (remaining intra prediction mode information). The decoding device may construct an MPM list. The MPM list is configured identically to the MPM list configured by the encoding device. That is, the MPM list may include the intra prediction mode of a surrounding block, or may further include specific intra prediction modes according to a predetermined method.
[0150] The decoding device can determine the intra prediction mode of the current block based on the MPM list and the intra prediction mode information. For example, when the value of the MPM flag is 1, the decoding device can derive the candidate pointed to by the MPM index among the MPM candidates in the MPM list as the intra prediction mode of the current block.
[0151] As another example, when the value of the MPM flag is 0, the decoding device can derive the intra prediction mode indicated by the remaining intra prediction mode information (which may be called the MPM remainder information) among the remaining intra prediction modes as the intra prediction mode of the current block.
[0152] The above intra prediction type may include, for example, various types related to non-regular intra prediction in addition to the regular intra prediction type. For example, it may include DIMD (decoder side intra mode derivation), TIMD (fusion for template-based intra mode derivation or template-based intra mode derivation), MIP (matrix-based intra sample prediction), intraTMP (intra template matching prediction), OBIC (occurrence-based intra coding), TMRL (template-based multiple reference line intra prediction), SGPM (spatial Geometric partitioning mode), EIP (extrapolation filter-based intra prediction) and / or NN_intra (neural network-based intra prediction), ISP (Intra sub-partitions), etc. Depending on the intra prediction type, some or all of the information regarding the above intra prediction mode may not be signaled separately.
[0153] The coding device derives surrounding reference samples of the current block (S310). The surrounding reference samples may include surrounding reference samples of the current block. The surrounding reference samples may be referred to as reference samples. The surrounding reference samples of the current block may include H+W samples located to the left of the current block of size WХH, W+H samples located to the top of the current block, and at least one sample adjacent to the top-left of the current block. Alternatively, the surrounding reference samples of the current block may include multiple rows of upper surrounding samples and multiple columns of left surrounding samples. When MRL or TMRL is applied, the number of surrounding reference samples may increase.
[0154] The coding device performs intra prediction on the current block to generate prediction samples (S330). The coding device can derive the prediction samples based on the intra prediction mode / type and the reference samples. The coding device can derive a reference sample according to the intra prediction mode of the current block among the reference samples of the current block, and can derive a prediction sample of the current block based on the reference sample.
[0155] The encoding device can derive a residual signal based on the above-mentioned prediction samples (prediction blocks) and signal it to the decoding device after undergoing a conversion / quantization procedure. The decoding device can restore a block / picture based on the above-mentioned prediction samples (prediction blocks) and residual samples (residual blocks).
[0156] Meanwhile, as described above, the prediction unit of the encoding / decoding device may perform inter-prediction on a block-by-block basis to derive prediction samples. Inter-prediction may represent a prediction derived in a manner dependent on data elements (i.e., sample values, or motion information, etc.) of picture(s) other than the current picture. When inter-prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) specified by a motion vector on the reference picture pointed to by the reference picture index. At this time, to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information of the current block can be predicted on a block, sub-block, or sample basis, based on the correlation of motion information between surrounding blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-prediction type information (L0 prediction, L1 prediction, Bi prediction, etc.). When inter-prediction is applied, neighboring blocks may include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture containing the reference blocks and the reference picture containing the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be referred to by names such as collocated reference block, collocated CU (colCU), etc., and the reference picture containing the temporal neighboring blocks may be referred to as collocated picture (colPic).For example, a list of motion information candidates may be constructed based on the surrounding blocks of the current block, and flag or index information indicating which candidate is selected (used) to derive the motion vector and / or reference picture index of the current block may be signaled. Inter-prediction may be performed based on various prediction modes; for example, in the case of skip mode and merge mode, the motion information of the current block may be the same as the motion information of the selected surrounding block. In the case of skip mode, unlike merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected surrounding block is used as a motion vector predictor, and the motion vector difference may be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.
[0157] The above motion information may include L0 motion information and / or L1 motion information depending on the inter-prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be called an L0 motion vector or MVL0, and a motion vector in the L1 direction may be called an L1 motion vector or MVL1. A prediction based on an L0 motion vector may be called an L0 prediction, a prediction based on an L1 motion vector may be called an L1 prediction, and a prediction based on both the L0 motion vector and the L1 motion vector may be called a pair (Bi) prediction. Here, the L0 motion vector may represent a motion vector associated with reference picture list L0 (L0), and the L1 motion vector may represent a motion vector associated with reference picture list L1 (L1). Reference picture list L0 may include pictures that are prior to the current picture in output order as reference pictures, and reference picture list L1 may include pictures that are subsequent to the current picture in output order. The aforementioned previous pictures may be referred to as forward (reference) pictures, and the aforementioned subsequent pictures may be referred to as reverse (reference) pictures. The reference picture list L0 may include additional pictures that are output later than the current picture as reference pictures. In this case, within the reference picture list L0, the previous pictures may be indexed first, and the subsequent pictures may be indexed next. The reference picture list L1 may include additional pictures that are output earlier than the current picture as reference pictures. In this case, within the reference picture list L1, the subsequent pictures may be indexed first, and the previous pictures may be indexed next. Here, the output order may correspond to the POC (picture order count) order.
[0158] The video / image encoding procedure based on inter prediction and the prediction unit within the encoding device may perform the following, for example, in a general manner for inter prediction.
[0159] Figure 4 illustrates an exemplary inter prediction procedure.
[0160] Referring to FIG. 4, as described above, the inter prediction procedure may include an inter prediction mode / type determination step, a motion vector derivation / refining step, and an inter prediction execution (prediction sample generation) step. The inter prediction procedure may be performed in an encoding device and a decoding device as described above. In this document, the term "coding device" may include an encoding device and / or a decoding device.
[0161] The coding device derives an inter prediction mode / type (S400). The encoding device can determine an inter prediction mode / type applied to the current block among the various inter prediction modes / types disclosed in this document and can generate prediction-related information. The prediction-related information may include inter prediction mode information indicating an inter prediction mode applied to the current block and / or inter prediction type information indicating an inter prediction type applied to the current block. The decoding device can determine an inter prediction mode / type applied to the current block based on the prediction-related information.
[0162] The coding device derives / modifies the motion vector of the current block (S410). The coding device can derive / modify the motion vector of the current block based on the determined inter-prediction mode / type. Here, to derive / modify the motion vector, motion information of the surrounding blocks of the current block may be used.
[0163] For example, when a skip mode or a merge mode is applied to the current block, the coding device may construct a merge candidate list and select one merge candidate from among the merge candidates included in the merge candidate list. Information indicating the selected merge candidate (e.g., merge index) may be included in the prediction-related information.
[0164] As another example, when an (A)MVP mode is applied to the current block, the decoding device may construct an (A)MVP candidate list and use the motion vector of a selected MVP candidate among the MVP (motion vector predictor) candidates included in the (A)MVP candidate list as the MVP of the current block. The selection may be indicated based on selection information (MVP flag or MVP index). In this case, information regarding the MVD as well as the selection information may be included in the prediction-related information.
[0165] Meanwhile, as described below, movement information of the current block can be derived without constructing a candidate list, and in this case, the movement information of the current block can be derived according to the procedure initiated in the prediction mode / type described below. In this case, the construction of the candidate list as described above may be omitted.
[0166] The coding device predicts the current block (generates prediction samples) based on the derived / modified motion vector (S420). The coding device can derive prediction samples of the current block using samples of the reference block pointed to by the motion vector on the reference picture.
[0167] The encoding device can derive residual samples based on the prediction samples. It can generate residual information based on the residual samples. The residual information may include information regarding quantized transform coefficients as described above. The encoding device encodes image information including prediction-related information and / or residual information. The encoding device can output the encoded image information in the form of a bitstream. The prediction-related information may include information related to the prediction procedure, such as prediction mode information and / or motion information.
[0168] The output bitstream can be stored in a (digital) storage medium and transmitted to a decoding device, or it can be transmitted to a decoding device via a network.
[0169] The decoding device performs inter-prediction based on prediction-related information. Based on the prediction-related information, the decoding device derives an inter-prediction mode / type for the current block, derives / modifies movement information of the current block, and generates prediction samples within the current block based on the intra-prediction mode / type and / or the movement information. In this case, the decoding device may perform a prediction sample filtering procedure. Prediction sample filtering may be referred to as post-filtering. Some or all of the prediction samples may be filtered by the prediction sample filtering procedure. In some cases, the prediction sample filtering procedure may be omitted.
[0170] The decoding device can derive residual samples for the current block based on the residual information. Specifically, the inverse quantization unit of the residual processing unit derives transform coefficients by performing inverse quantization based on the quantized transform coefficients derived from the residual information, and the inverse transform unit of the residual processing unit derives residual samples for the current block by performing an inverse transform on the transform coefficients. The decoding device can generate restored samples for the current block based on the predicted samples and / or the residual samples, and derive a restored block containing the restored samples.
[0171] In the embodiments of the present invention, the order of each process in the examples of all block diagrams may be changed or omitted.
[0172] In the following, the terms "prediction block," "prediction unit block," and "decoding block" may be used interchangeably with the same meaning.
[0173] In the following, the terms "luminance component block" and "luminance block" may be used interchangeably with the same meaning.
[0174] In the following, the terms "color difference component block" and "color difference block" may be used interchangeably with the same meaning.
[0175] Figure 5 shows examples of intra-prediction-based video / image encoding methods.
[0176] Referring to FIG. 5, S500 can be performed by the prediction unit of the encoding device, S505 can be performed by the residual processing unit of the encoding device, and S510 or S515 can be performed by the entropy encoding unit of the encoding device. Specifically, the prediction-related information can be derived by the prediction unit and encoded by the entropy encoding unit. The residual information can be derived by the residual processing unit and encoded by the entropy encoding unit. The residual information is information regarding the residual samples. The residual information may include information regarding quantized transformation coefficients for the residual samples. As described above, the residual samples are derived into transformation coefficients through the transformation unit of the encoding device, and the transformation coefficients can be derived into quantized transformation coefficients through the quantization unit. Information regarding the quantized transformation coefficients can be encoded in the entropy encoding unit through a residual coding procedure.
[0177] The encoding device performs intra prediction for the current block (S500). The encoding device derives an intra prediction mode / type for the current block, derives reference samples for the current block, and generates prediction samples within the current block based on the intra prediction mode / type and the reference samples. Here, the procedures for determining the intra prediction mode / type, deriving surrounding reference samples, and generating prediction samples may be performed simultaneously, or any one procedure may be performed before another. The encoding device can determine the mode / type applied to the current block among a plurality of intra prediction modes / types. The encoding device can compare the RD costs for the intra prediction modes / types and determine the optimal intra prediction mode / type for the current block.
[0178] Meanwhile, the encoding device may perform a predictive sample filtering procedure. Predictive sample filtering may be referred to as post-filtering. Some or all of the predicted samples may be filtered by the predictive sample filtering procedure. In some cases, the predictive sample filtering procedure may be omitted.
[0179] The encoding device generates residual samples for the current block based on predicted samples (S505). The encoding device can compare the predicted samples with the original samples of the current block based on phase and derive the residual samples.
[0180] The encoding device can encode image / video information including information regarding the intra prediction (prediction-related information) and / or information regarding the residual samples (residual information) (S510 or S515). The prediction-related information may include the intra prediction mode information and the intra prediction type information. The encoding device may output the encoded image / video information in the form of a bitstream. The output bitstream may be transmitted to a decoding device via a storage medium or a network.
[0181] The above residual information may include residual coding syntax elements. The encoding device may transform / quantize the above residual samples to derive quantized transformation coefficients. The above residual information may include information regarding the quantized transformation coefficients.
[0182] Meanwhile, as described above, the encoding device can generate a restored picture (including restored samples and a restored block). To this end, the encoding device can derive (modified) residual samples by performing inverse quantization / inverse transform processing on the quantized transform coefficients again. The reason for performing inverse quantization / inverse transform on the residual samples after transforming / quantizing them in this manner is to derive residual samples identical to those derived from the decoding device as described above. Based on the predicted samples and the (modified) residual samples, the encoding device can generate a restored block containing restored samples for the current block. Based on the restored block, a restored picture for the current picture can be generated. As described above, an in-loop filtering procedure, etc., may be further applied to the restored picture.
[0183] The decoding device can perform an operation corresponding to the operation performed by the encoding device. A video / image decoding procedure based on intra-prediction may include, for example, the following.
[0184] Figure 6 shows examples of intra-prediction-based video / image decoding methods.
[0185] Referring to FIG. 6, S600 can be performed by the entropy decoding unit of the decoding device, S610 can be performed by the prediction unit of the decoding device, S615 can be performed by the residual processing unit of the decoding device, and S620 can be performed by the adder or restoration unit of the decoding device.
[0186] Specifically, the decoding device obtains image / video information from the bitstream (S600). The image / video information may include prediction-related information and / or residual information.
[0187] The decoding device performs intra-prediction based on prediction-related information (S610). The decoding device can derive an intra-prediction mode / type for the current block based on the prediction-related information, derive reference samples for the current block, and generate prediction samples within the current block based on the intra-prediction mode / type and the reference samples. In this case, the decoding device can perform a prediction sample filtering procedure. Prediction sample filtering may be referred to as post-filtering. Some or all of the prediction samples may be filtered by the prediction sample filtering procedure. In some cases, the prediction sample filtering procedure may be omitted.
[0188] The decoding device performs residual processing based on residual information (S615). The decoding device can derive residual samples for the current block based on the residual information. Specifically, the inverse quantization unit of the residual processing unit derives transformation coefficients by performing inverse quantization based on the quantized transformation coefficients derived based on the residual information, and the inverse transformation unit of the residual processing unit derives residual samples for the current block by performing inverse transformation on the transformation coefficients.
[0189] The decoding device generates a restored block / picture (S620). The decoding device can generate restoration samples for the current block based on the prediction samples and / or the residual samples, and derive a restored block containing the restoration samples. A restored picture for the current picture can be generated based on the restored block. As described above, an in-loop filtering procedure, etc., may be further applied to the restored picture.
[0190] The intra prediction mode information and / or the intra prediction type information may be encoded / decoded through the binarization and coding methods described in this disclosure. For example, the intra prediction mode information and / or the intra prediction type information may be binarized through fixed-length binarization, truncated rice binarization, truncated univariate binarization, etc. For example, the intra prediction mode information and / or the intra prediction type information may be encoded / decoded through entropy coding (e.g., CABAC, CAVLC).
[0191] Figure 7 shows examples of directional intra prediction modes.
[0192] Referring to FIG. 7, directional intra prediction modes may include directional intra prediction modes from Mode #2 to Mode #65. In this case, Mode #50 may represent a vertical intra prediction mode, and Mode #18 may represent a horizontal intra prediction mode. Meanwhile, as an example, the number and type of candidate intra prediction modes may be changed. Meanwhile, one or more non-directional intra prediction modes may be considered, for example, Mode #0 may represent an intra planner prediction mode (planner mode), and Mode #1 may represent an intra DC prediction mode (DC mode).
[0193] For intra-prediction modes, prediction mode numbers can be assigned, for example, as shown in the following table.
[0194]
[0195] Figure 8 exemplarily illustrates the prediction directions associated with wide-angle intra prediction modes.
[0196] Referring to FIG. 8, the intra prediction modes may include wide-angle intra prediction modes in addition to the 65 directional intra prediction modes described above. That is, the prediction directions indicated by dashed lines may be related to wide-angle intra prediction modes that apply only to non-square blocks. For example, a wide-angle intra prediction mode may be used when the current block is a non-square block.
[0197] When a wide-angle intra prediction mode is applied, information about the existing intra prediction mode is signaled, and after the information about the intra prediction mode is parsed, the information about the intra prediction mode can be remapped to the index of the wide-angle intra prediction mode. Accordingly, the total number of intra prediction modes for a specific block (e.g., a non-square block of a specific size) may not change, that is, the total number of intra prediction modes is 67, and the intra prediction mode coding for the specific block may not change.
[0198] Meanwhile, according to one embodiment of the present disclosure, a TIMD technique may be applied. The TIMD may refer to template-based intra-mode derivation or fusion of template-based intra-mode derivation. The TIMD technique may be referred to as a TIMD type. In TIMD, a template including surrounding samples of the current block is derived, a predictor of the template according to a candidate prediction mode is derived based on the reference samples of the template (i.e., prediction samples of the template are derived), and the optimal intra-prediction mode can be derived by comparing this with the restored template (i.e., restored samples of the template).
[0199] TIMD derives an intra-frame prediction mode through a template and performs intra-frame prediction using the derived prediction mode. At this time, to improve the accuracy of the prediction mode, a method can be used to derive the intra-frame prediction mode of the target block by minimizing the difference between the encoded template area and the predicted prediction template area using reference samples for the template area. At this time, to quantify the difference between the prediction template area and the re-encoded template area, according to the embodiment, one of various cost functions such as SATD (Sum of absolute transformed differences) or SAD (Sum of Absolute Differences) is selectively applied to calculate the cost for each prediction mode, and one or more prediction modes with low costs are determined. Then, one or more prediction blocks are generated using the determined prediction modes, and if there are multiple prediction blocks, a weighted sum is performed to generate a first prediction block. According to the embodiment, if necessary for the accuracy of the prediction, pixel values may be changed on a pixel-by-pixel basis for the first prediction block, and this can be performed using at least one of the mode information of the prediction blocks used to generate the first prediction block, the pixel values of the prediction blocks, and the locations of the prediction pixels within the prediction block.
[0200] According to the TIMD technique, n intra prediction modes having the minimum SAD cost or minimum SATD cost can be derived from the above candidate intra prediction modes. Here, n can be 1 or 2 or greater. For example, if n is 2, two TIMD derived modes can be weighted-based fused after applying PDPC. The weights can be calculated based on the cost for the n modes.
[0201] The weights for TIMD can be calculated, for example, as follows:
[0202]
[0203]
[0204] When deriving weights, division operations can be replaced using an LUT.
[0205] Meanwhile, the TIMD derivation mode (TIMD-based mode) may derive only one intra prediction mode when certain conditions are satisfied. For example, if the cost of an intra prediction mode having the minimum SAD cost or minimum SATD cost calculated based on TIMD is smaller than a threshold, only one intra prediction mode may be derived. As another example, if the difference between the first SATD cost of a first intra prediction mode having the minimum SATD cost calculated based on TIMD and the second SATD cost of a second intra prediction mode having the second minimum SATD cost is greater than a threshold, only one intra prediction mode may be derived.
[0206] Meanwhile, the example described above illustrates how to calculate costs based on SAD or SATD, but this is just an example, and various cost functions including MR SAD (Mean-Removed Sum of Absolute Differences) may be used.
[0207] Meanwhile, TIMD may not be used with ISP or MRL. For example, if TIMD is applied, ISP may not be used, and if TIMD is applied, MRL may not be used.
[0208] Meanwhile, to reduce the computational complexity for TIMD and increase signaling efficiency, a TIMD merge mode may be used. The TIMD merge mode may be referred to as a TIMD merge type. In this case, the TIMD derivation mode(s) of the current block may be derived based on a previously TIMD-coded block. To this end, adjacent and non-adjacent spatial surrounding blocks may be scanned. The spatial surrounding blocks may be located in at least one of the left, upper, lower-left, upper-right, and upper-left directions of the current block. Additionally, temporal surrounding blocks may be scanned for the TIMD merge mode. The temporal surrounding blocks may include at least one of the lower-right block, center-lower-right block, right block, and lower block of the same-location block of the reference picture (call picture). If a scanned block is coded in TIMD or TIMD merge mode, a TIMD merge candidate containing TIMD information of the scanned block (e.g., prediction mode, fusion flag, fusion weight, etc.) may be added to the TIMD merge (candidate) list. The TIMD merge list may contain n candidates, where n may be signaled via HLS such as SPS, PPS, etc., or determined as a specific value (e.g., 5, etc.).
[0209] Whether to apply TIMD merge mode may be signaled based on a flag (timd merge flag) or implicitly determined based on a predetermined condition. The predetermined condition may include whether the size of the current block is within a predetermined range. The predetermined condition may include whether the current block is a square block.
[0210] The above TIMD merge flag can be signaled after the TIMD flag signaling when the TIMD flag indicates true (1).
[0211] The above TIMD merge list can be configured, for example, as shown in the following table.
[0212]
[0213] As described above, the TIMD merge list may include TIMD merge candidates. The TIMD merge candidates may include n intra prediction modes (IPMs) and weights.
[0214] The candidates in the above TIMD merge list can be sorted based on SAD cost.
[0215] Figure 9 shows an example of the TIMD technique.
[0216] Referring to FIG. 9, the template or template area of the block to be predicted (i.e., the current block) may include a left template and a top template. The left template may be located around the left side of the block to be predicted, and the top template may be located around the top side of the block to be predicted. The reference area of the template may be located around the template area. The reference samples of the template may include first reference samples of the left reference area and second reference samples of the top reference area. The left reference area may be located around the left side of the left template, and the top reference area may be located around the top side of the top template. The width of the left template may be a, the height of the top template may be b, the width of the top reference area of the template may be c, and the height of the left reference area may be d.
[0217] Specifically, at least one of the lengths of a, b, c, and d can be adaptively determined. At least one of the lengths of a, b, c, and d can be implicitly determined based on the width or height value of the block to be predicted, by checking whether the template area and the reference area of the template are valid, and the template area can be determined in the coding device by parsing one or more flags or indices for determining the width or height of the template.
[0218] Meanwhile, TIMD can determine whether to perform TIMD by parsing a 1-bit flag (timd_flag) indicating whether to perform TIMD. In this case, the parsing of timd_flag can be determined based on the TIMD enable / enable flag (on / off flag), the size of the current block, the prediction mode of the current block, whether the current block is a luminance component, whether the current block is a chrominance component, etc. The above TIMD enable / enable flag (on / off flag) can be called the TIMD enable flag and can be parsed in higher-level syntax such as SPS (sequence parameter set) or PPS (picture parameter set).
[0219] For example, if the parsed timd_flag is 1, a TIMD prediction block can be generated.
[0220] Figure 10 shows an example of a method for generating a TIMD prediction block.
[0221] Referring to FIG. 10, the prediction unit of the coding device may include a prediction mode determination unit (1000), a prediction block generation unit (1010) for each mode, a first weighted sum execution unit (1020) for each group, and a second weighted sum execution unit (1030) based on pixel location.
[0222] The prediction mode determination unit (1000) can determine the intra prediction mode(s) used for the prediction of the current block among the candidate intra prediction modes. The prediction mode determination unit (1000) can use directional prediction modes and / or non-directional prediction modes as candidate intra prediction modes. Additionally, the prediction mode determination unit (1000) can select up to n intra prediction modes to be used for the generation of the TIMD prediction block among the candidate intra prediction modes.
[0223] The mode-specific prediction block generation unit (1010) can generate prediction block(s) for the current block according to the prediction mode(s) selected by the prediction mode determination unit (1000). The prediction block(s) may be predicted block(s) by prediction mode or intermediate prediction block(s).
[0224] The group-specific first weighted sum execution unit (1020) performs a weighted sum for prediction blocks for each group of prediction modes. When multiple prediction modes are applied within the same group, the group-specific first weighted sum execution unit (1020) can perform a weighted sum for the prediction blocks for each group. The prediction block(s) for each prediction mode generated by the mode-specific prediction block generation unit (1010) can be obtained, and after classifying the prediction blocks based on the group of each prediction mode, a group-specific prediction block can be generated by performing a weighted sum on the prediction blocks within one group based on the cost calculated by the prediction mode determination unit (1000). The group-specific prediction block may also be referred to as an intermediate prediction block.
[0225] The pixel location-based second weighted sum execution unit (1030) can obtain prediction blocks for each group generated by the first weighted sum execution unit when there is an upper or left group, and can perform a weighted sum by applying weights according to pixel location based on this.
[0226] Meanwhile, each of the prediction block by prediction mode, the prediction block by group, the intermediate prediction block, and the (final) prediction block may be different from or the same as each other.
[0227] Figure 11 shows an example of a method for determining the intra-prediction mode(s) to be used for prediction.
[0228] Referring to FIG. 11, the prediction mode determination unit (1000) may include a search mode determination unit (1100), a cost calculation unit per mode (1110), a group determination unit per mode (1120), and a number of modes to use determination unit (1130).
[0229] The search mode determination unit (1100) can determine which intra-prediction mode(s) to search for in the prediction mode determination unit (1000). For example, it can determine whether to search for prediction modes defined according to the encoder / decoder agreement, or to search for prediction modes within the MPM list, or / and search for non-directed modes. The non-directed mode may be a Planar mode or / and DC mode, etc., and the defined prediction mode may be determined based on the block size and aspect ratio, etc.
[0230] The cost calculation unit (1110) for each prediction mode can calculate the cost for each prediction mode for the prediction mode(s) decided to be searched by the search mode determination unit (1100). The prediction mode(s) decided to be searched may be called candidate intra prediction mode(s). For example, the cost for each prediction mode can be calculated by using the reference area of the template to perform predictions on the upper template and the left template based on the candidate intra prediction mode, then comparing them with the previously restored left and upper template areas to obtain the cost of the upper template and the cost of the left template, respectively, and adding them together. Alternatively, the cost for each mode can be calculated based on the (weighted) average of the cost of the upper template and the cost of the left template.
[0231] Meanwhile, as an example, when DC mode is applied, the reference samples of the left template for predicting the left template may consist of left reference samples adjacent to the left template. In this case, the average value (or DC value) of the left reference samples adjacent to the left template may be used for predicting the left template. Additionally, when DC mode is applied, the reference samples of the upper template for predicting the upper template may consist of upper reference samples adjacent to the upper template. In this case, the average value (or DC value) of the upper reference samples adjacent to the upper template may be used for predicting the upper template.
[0232] As another example, when DC mode is applied, the average value (or DC value) using left reference samples adjacent to the left template and top reference samples adjacent to the top template may be used as the predicted value for both the left template and the top template.
[0233] The mode-specific group determination unit (1120) can determine a group indicating which direction template the prediction mode to be searched is more influenced by, for the prediction modes decided to be searched by the search mode determination unit (1100). Alternatively, the mode-specific group determination unit (1120) may determine a group only for the determined n prediction modes once the number of prediction modes to be used is determined. In this case, the mode-specific group determination unit (1120) may perform the procedure after the mode number determination unit (1130).
[0234] For example, groups by mode can be derived into m groups by comparing the cost of the top template with the cost of the left template. For example, groups can be derived based on the ratio of the costs of the two templates or the magnitude of the difference.
[0235] Specifically, the cost of the top template is cost A , the cost of the left template is cost L , Let be any positive real number for group classification, and groups can be derived through conditions as shown in the following table. This example is an example where the groups are divided into three: the top group, the left group, and the non-directed group.
[0236]
[0237] At this time, group classification criteria (parameters) For example, it can be predefined, or, for another example, it can be changed depending on the aspect ratio of the block to be predicted or / and the size of the block to be predicted.
[0238] Meanwhile, in the directional prediction mode, groups can be derived by comparing the costs of the left template and the top template. In the non-directional prediction mode, groups can be implicitly derived as non-directional, or groups can be derived by comparing the costs of the left template and the top template.
[0239] The mode number determination unit (1130) can determine up to n prediction modes to be used for TIMD prediction. The mode number determination unit (1130) may be omitted depending on the case. For example, the number of prediction modes to be used for TIMD prediction can be determined up to n by selecting candidate intra prediction modes in order of lowest cost or / and selecting A directional modes in order of lowest cost and B non-directional modes in order of lowest cost, and calculating the cost difference for each prediction mode.
[0240] Specifically, the cost of the directional mode di , the cost of the non-directional mode cost nj and, , Assuming that is an arbitrary positive real number used to determine the number of prediction modes to use, groups can be derived through the conditions shown in the following table. This example describes a case where two directional modes are selected in order of lowest cost and one non-directional mode in order of lowest cost, and the number of prediction modes to be used for TIMD forecasting is determined to be up to three. Lower numbers for i and j correspond to lower costs.
[0241]
[0242] At this time, classification criteria for the number of predicted usage modes , For example, it can be predefined, or, for another example, it can be changed depending on the aspect ratio of the block to be predicted or / and the size of the block.
[0243] Figure 12 shows an example of generating prediction blocks by group.
[0244] Referring to FIG. 12, group-specific prediction blocks can be generated by performing a weighted sum on prediction blocks generated based on each prediction mode within each group. This example is a case where, when two intra-prediction modes are derived into the same group, a group prediction block is generated by performing a weighted sum on prediction blocks generated based on each intra-prediction mode. The weighted sum step for generating group-specific prediction blocks can be performed in a group-specific first-order weighted sum execution unit.
[0245] For example, the weights for prediction blocks by prediction mode within each group can be calculated using the following mathematical formula based on the costs by prediction mode calculated by the prediction mode determination unit. The weights are proportional to the sum of the costs of the remaining prediction modes within the group, excluding the prediction mode of the group itself, and can be adjusted through normalization so that the sum of the total weights becomes a fixed value.
[0246]
[0247] Here wi is the weight for the prediction block for each prediction mode within a group, and K is the number of prediction modes within a group.
[0248] Meanwhile, as an example of reducing the complexity of the division operation when calculating weights, a logarithmic operation or / and a shift operation or / and a look-up table (LUT) may be used.
[0249] For example, if the number of prediction modes within a group is 2 and the total sum of weights is adjusted to 64, the weights can be calculated according to an embodiment such as the following table. The values used in the following embodiment are not limited and can be changed to other values.
[0250]
[0251] Here, DivSigTable
[0016] is the relative size (2) of the sum of the costs (S1) of the two prediction modes within the group. x S1 2 x-1 It is an LUT that stores 1 / S1 approximated as a 4-bit integer according to the position within the range of . The size of the LUT and the value of the total sum of weights for approximation are not limited and may vary depending on the approximation accuracy for the weights and may be determined based on K, which is the number of prediction modes within one group, and / or the size of the total sum of costs of the prediction modes within the group, and / or the size of the block.
[0252] Meanwhile, the following mathematical formula is the size of the table (LUT) according to the number of prediction modes K within the group n ) and weighted sum (T k This is an example of calculating ).
[0253]
[0254] Here, the values of a, b, c, and d that determine the size of the table may change depending on the approximation accuracy. The above a, b, c, and d may be positive integers.
[0255] As another example, if the number of prediction modes within a group is 3 and the total sum of weights is adjusted to 64, the weight can be assigned first to the prediction mode with the lowest cost so that the sum of the remaining weights becomes a power of 2. The weight for the prediction mode with the lowest cost can be determined based on the total sum of the costs of the three prediction modes and the sum of the costs of the remaining two prediction modes. The calculation of the weights for the remaining two prediction modes can be performed in the same way as the weight calculation method when the number of prediction modes within a group is 2. Meanwhile, the total sum of weights, 64, may be 128, 256, or other values as examples.
[0256] Figure 13 shows an example of weight settings for prediction blocks by group when performing a pixel location-based weighted sum.
[0257] Referring to FIG. 13, the weights for the prediction blocks by group can be set to change at regular intervals while maintaining the same value in single or multiple line units. The way the weights change in single or multiple line units may, for example, change linearly, for another example, change non-linearly, or for another example, change in a stepwise manner.
[0258] For example, FIG. 13(a) and FIG. 13(c) are examples of cases where weights are set to change according to pixel position (i.e., every line), and FIG. 13(b) and FIG. 13(d) are examples of cases where weights are set to change at regular intervals while maintaining the same value every two lines. In this case, the way weights change can be linear, non-linear, or stepwise. For example, when weights change every line, the weights may change linearly, and when weights change stepwise at regular intervals, the weights may change non-linearly.
[0259] Meanwhile, the amount of change in weights can be predefined, for example, or calculated as a linear or non-linear function based on pixel location information or / and the sum of costs of prediction modes within another group or / and the difference between the sum of costs of prediction modes within the current group and the sum of costs of prediction modes within another group or / and the block size or / and the aspect ratio of the block.
[0260] Weights for prediction blocks by group can be assigned higher the closer they are to the reference template in that direction.
[0261] Figure 14 shows an example of performing a pixel-location-based weighted sum.
[0262] Referring to FIG. 14, a pixel-position-based weighted sum can be performed on the prediction block(s) for each group. This example is one in which three group classifications (top group, left group, and non-directed group) are formed, weights that change for each line are used, and normalization is performed such that the sum of all weights for a specific phase (or specific sample position) is 64. Specifically, FIG. 14 (a) is an example in which the top group prediction block and the non-directed group prediction block are generated as a result of the first-order weighted sum execution unit for each group, FIG. 14 (b) is an example in which the top group prediction block, the left group prediction block, and the non-directed group prediction block are generated as a result of the first-order weighted sum execution unit for each group, and FIG. 14 (c) is an example in which only the top group prediction block is generated as a result of the first-order weighted sum execution unit for each group.
[0263] Meanwhile, the normalization value may be predefined, for example, or changed based on the number of groups and / or the number of prediction modes to use and / or the equal weight line length and / or the size of the block and / or the aspect ratio of the block, for other examples.
[0264] Pixel location-based weighted sum can be performed in the pixel location-based second weighted sum execution unit.
[0265] Figures 15a and 15b show examples of cost calculation based on the directionality of the prediction mode.
[0266] Referring to FIG. 15a and FIG. 15b, when calculating the cost per prediction mode in the mode-specific cost calculation unit, the cost of the prediction mode can be calculated based on the directionality of the prediction mode or the prediction mode number.
[0267] For example, as shown in FIG. 15a, if the prediction mode to be explored has a horizontal direction or is close to a horizontal prediction mode, the cost can be calculated based on the cost of the left template. As another example, as shown in FIG. 15b, if the prediction mode to be explored has a vertical direction or is close to a vertical prediction mode, the cost can be calculated based on the cost of the top template.
[0268] For example, if the difference between the prediction mode number of the (candidate) prediction mode and the prediction mode number of the horizontal prediction mode is K or less, the (candidate) prediction mode may be determined to have a horizontal directionality or to be close to the horizontal prediction mode. As another example, if the difference between the prediction mode number of the (candidate) prediction mode and the prediction mode number of the vertical prediction mode is K or less, the (candidate) prediction mode may be determined to have a vertical directionality or to be close to the vertical prediction mode. K may represent the absolute value of the difference between the prediction mode number of the corresponding prediction mode and the prediction mode number of the horizontal or vertical prediction mode.
[0269] Here, the prediction mode number range K of the vertical prediction mode or the prediction mode close to the horizontal prediction mode may be predetermined, for example, or, for another example, changed according to the aspect ratio of the block to be predicted or / and the size of the block to be predicted. For example, when determining the range of the prediction mode close to the horizontal prediction mode, the range K may become smaller or larger as the width of the block becomes greater than the height. For another example, when determining the range of the prediction mode close to the vertical prediction mode, the range K may become smaller or larger as the height of the block becomes greater than the width.
[0270] Meanwhile, as in the embodiment described above, by calculating the cost of the prediction mode based on the directionality of the prediction mode, more efficient computation can be enabled and intra-prediction performance can be improved.
[0271] Figure 16 shows an example of cost calculation based on prediction mode directionality and template weights.
[0272] Referring to FIG. 16, when calculating the cost per mode in the mode-specific cost calculation unit, the cost of the corresponding prediction mode can be calculated based on the directionality of the prediction mode or the prediction mode number and the template weight. For example, the prediction mode number can be divided into J intervals, and the cost of the corresponding prediction mode can be calculated by weighting the cost of the top template and / or the left template according to each interval. The range and template weight of each prediction mode number interval can be predefined, for example, or, for another example, changed according to the aspect ratio of the block to be predicted or / and the size of the block to be predicted.
[0273] For example, this example is one where the prediction mode number is divided into 5 intervals. Specifically, the cost per mode can be calculated based on the following mathematical formula.
[0274]
[0275] In this case, the template weights can be set based on the following table.
[0276]
[0277] If the (candidate) prediction mode is an undirected prediction mode, it may fall within the first interval of the table above. In this case, the upper template weight may be 2 and the left template weight may be 2. Meanwhile, the upper template weight may be applied to the cost of the upper template, and the left template weight may be applied to the cost of the left template.
[0278] As another example, template weights can be derived based on interval information to which the prediction mode number of the prediction mode belongs. The interval information can be represented based on first group information or a first group index.
[0279] Meanwhile, in FIG. 16, prediction mode numbers 67 to 80 and prediction modes -1 to -14 (i.e., prediction modes located clockwise in the upward-to-right diagonal direction and prediction modes located counterclockwise in the downward-to-left diagonal direction) may represent wide-angle intra prediction modes derived according to WAIP (wide angular intra prediction).
[0280] For example, at least one of the prediction modes of prediction mode numbers -1 to -14 may belong to the same interval as at least one of the prediction modes of prediction mode numbers 2 to 23. At least one of the prediction modes of prediction mode numbers 67 to 80 may belong to the same interval as at least one of the prediction modes of prediction mode numbers 46 to 66.
[0281] Meanwhile, as in the embodiment described above, by calculating the cost of the prediction mode based on the directionality of the prediction mode, more efficient computation can be enabled and intra-prediction performance can be improved.
[0282] Figures 17a and 17b show examples of group classification correction.
[0283] Referring to FIG. 17a and FIG. 17b, when deriving a group by mode in the group by mode determination unit, the group of the predicted mode can be derived based on the directionality of the predicted mode. Alternatively, the group of the predicted mode can be changed.
[0284] For example, if a prediction mode has a horizontal directionality or is close to a horizontal prediction mode, the group corresponding to that prediction mode may be derived as a left group or a non-directional group. In this case, the top group may not be considered.
[0285] For example, if the prediction mode has a horizontal direction or is close to a horizontal prediction mode and the cost of the top template is calculated low, and the mode direction and group classification differ, the group for that prediction mode can be derived as, for example, a top group or a non-directional group based on the cost of the top template or the left template. Alternatively, as another example, the group can be changed to a non-directional group.
[0286] For example, if the prediction mode has a horizontal directionality or is close to a horizontal prediction mode and the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a first threshold, the group of the prediction mode may be derived as either an upper group or a non-directional group based on whether the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a second threshold. Meanwhile, the first threshold and the second threshold may be any positive real numbers.
[0287] For example, if a prediction mode has a vertical orientation or is close to a vertical prediction mode, the group corresponding to that prediction mode may be derived as an upper group or a non-directional group. In this case, the left group may not be considered.
[0288] For example, if the prediction mode has a vertical orientation or is close to a vertical prediction mode and the cost of the left template is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode can be derived as a left group or a non-directional group based, for example, on the cost of the top template or the left template. Alternatively, as another example, the group can be changed to a non-directional group.
[0289] For example, if the prediction mode has a vertical directionality or is close to a vertical prediction mode and the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a first threshold, the group of the prediction mode may be derived as either a left group or a non-directional group based on whether the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a second threshold. Meanwhile, the first threshold and the second threshold may be any positive real numbers.
[0290] Specifically, FIG. 17a illustrates an example where, when the prediction mode has horizontal directionality, it is derived into an upper group or a non-directional group based on the cost of the upper template and the cost of the left template. For example, even if the prediction mode has horizontal directionality and the cost of the upper template is low, and thus it is derived into the upper group according to the group determination method described above, the group can be maintained as the upper group or changed to the non-directional group by determining whether a specific condition is satisfied based on a comparison of the cost of the upper template and the cost of the left template. In other words, when the prediction mode has horizontal directionality, it can be derived into either the upper group or the non-directional group based on a comparison of the cost of the upper template and the cost of the left template. In this case, if the cost of the left template is greater than a certain ratio or difference from the cost of the upper template (i.e., equal or greater), the prediction mode can be derived into the non-directional group, and in other cases, the prediction mode can be derived into the upper group.
[0291] Meanwhile, the above specific conditions may include the conditions in the following table.
[0292]
[0293] Here, the cost of the top template is cost A and the cost of the left template is cost L And, can be any positive real number for group classification. Any positive real number for group classification (parameter) For example, it can be predetermined, or, for another example, it can be changed depending on the aspect ratio of the block to be predicted or / and the size of the block to be predicted. Any positive real number (parameter) for the above group classification is the aforementioned group classification criteria (parameters) It could be the same as, or it could be different.
[0294] Specifically, FIG. 17b illustrates an example where, when the prediction mode has vertical directionality, a left group or a non-directional group is derived based on the cost of the top template and the cost of the left template. For example, even if the prediction mode has vertical directionality and the cost of the left template is low, resulting in a left group according to the group determination method described above, the group can be maintained as a left group or changed to a non-directional group by determining whether a specific condition is satisfied based on a comparison of the cost of the top template and the cost of the left template. In other words, when the prediction mode has vertical directionality, one of the groups, either a left group or a non-directional group, can be derived based on a comparison of the cost of the top template and the cost of the left template. In this case, if the cost of the top template is greater than a certain ratio or difference than the cost of the left template (i.e., equal or greater), the prediction mode can be derived as a non-directional group, and in other cases, the prediction mode can be derived as a left group.
[0295] Meanwhile, the above specific conditions may include the conditions in the following table.
[0296]
[0297] Here, the cost of the top template is cost A and the cost of the left template is cost L And, can be any positive real number for group classification. Any positive real number for group classification (parameter) For example, it can be predetermined, or, for another example, it can be changed depending on the aspect ratio of the block to be predicted or / and the size of the block to be predicted. Any positive real number (parameter) for the above group classification is the aforementioned group classification criteria (parameters) It could be the same as, or it could be different.
[0298] For example, if the difference between the prediction mode number of a prediction mode and the prediction mode number of a horizontal prediction mode is K or less, the prediction mode may be determined to have a horizontal direction or to be close to a horizontal prediction mode. As another example, if the difference between the prediction mode number of a prediction mode and the prediction mode number of a vertical prediction mode is K or less, the prediction mode may be determined to have a vertical direction or to be close to a vertical prediction mode. K may represent the absolute value of the difference between the prediction mode number of the corresponding prediction mode and the prediction mode number of the horizontal or vertical prediction mode.
[0299] Here, the prediction mode number range K of the vertical prediction mode or the prediction mode close to the horizontal prediction mode may be predetermined, for example, or, for another example, changed according to the aspect ratio of the block to be predicted or / and the size of the block to be predicted. For example, when determining the range of the prediction mode close to the horizontal prediction mode, the range K may become smaller or larger as the width of the block becomes greater than the height. For another example, when determining the range of the prediction mode close to the vertical prediction mode, the range K may become smaller or larger as the height of the block becomes greater than the width.
[0300] Meanwhile, information regarding group classification can be represented as second group information or a second group index.
[0301] Meanwhile, the accuracy of the final prediction result can be increased by deriving groups or correcting group classification using the directionality of the prediction mode, and the prediction quality can be improved by mitigating the discrepancy between the directionality of the prediction mode and the directionality of group classification.
[0302] Figure 18 shows an example of a TIMD technique with a different template configuration.
[0303] Referring to FIG. 18, the template or template area of the block to be predicted (i.e., the current block) may include top-left, top-right, top-left, and bottom-left templates. Based on this, embodiments related to the cost calculation by mode, group determination by mode, and group classification correction described above may be performed. In this case, the cost of the left template or the cost of the top template described above may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively. Meanwhile, the left template may include top-left and bottom-left templates, and the top template may include top-left and top-right templates.
[0304] The top-left and bottom-left templates may be located around the left side of the block to be predicted, and the top-left and top-right templates may be located around the top side of the block to be predicted. The reference area of the template may be located around the template area. The reference samples of the template may include first reference samples of the left reference area and second reference samples of the top reference area. The left reference area may be located around the left side of the top-left and bottom-left templates, and the top reference area may be located around the top side of the top-left and top-right templates. The width of the top-left and bottom-left templates may be a, and the height of the top-left and top-right templates may be b; the width of the top reference area of the template may be c, and the height of the left reference area may be d.
[0305] Specifically, at least one of the lengths of a, b, c, and d can be adaptively determined. At least one of the lengths of a, b, c, and d can be implicitly determined based on the width or height value of the block to be predicted, by checking whether the template area and the reference area of the template are valid, and the template area can be determined in the coding device by parsing one or more flags or indices for determining the width or height of the template.
[0306] For example, when calculating the mode-specific cost in the mode-specific cost calculation unit, the cost can be obtained by performing predictions on the top-left, top-right, top-left, and bottom-left template areas respectively using the template's reference area, and then comparing these predictions with the previously restored template areas to calculate the cost; or, as another example, by applying a cost function to calculate the cost and adding them up. Alternatively, it can be calculated based on the (weighted) average of the costs.
[0307] Meanwhile, as an example, when DC mode is applied, the reference samples of the left template for predicting the top-left and bottom-left templates may consist of left reference samples adjacent to the said top-left and bottom-left templates. In this case, the average value (or DC value) of the left reference samples adjacent to the said left template may be used for predicting the said top-left and bottom-left templates. Additionally, when DC mode is applied, the reference samples of the top template for predicting the top-left and top-right templates may consist of top reference samples adjacent to the said top-left and top-right templates. In this case, the average value (or DC value) of the top reference samples adjacent to the said top template may be used for predicting the said top-left and top-right templates.
[0308] As another example, when DC mode is applied, the average value (or DC value) using left reference samples adjacent to the top-left and bottom-left templates and top reference samples adjacent to the top-left and top-right templates may be used as the predicted value for all of the top-left template, the bottom-left template, the top-left template, and the top-right template.
[0309] For example, when determining mode-specific groups in the mode-specific group determination unit, the mode-specific groups can be derived into m groups by comparing the sum of the costs of the top-left template and the top-right template with the sum of the costs of the top-left template and the bottom-left template. As another example, groups can be derived based on the magnitude of the difference between the sum of the costs of the top-left template and the top-right template and the sum of the costs of the top-left template and the bottom-left template.
[0310] As another example, when calculating the cost per mode, the cost of the prediction mode can be calculated based on its directionality or prediction mode number. For instance, if the prediction mode to be explored has a horizontal directionality or is close to a horizontal prediction mode, the cost can be calculated based on the costs of the top-left and bottom-left templates. As another example, if the prediction mode to be explored has a vertical directionality or is close to a vertical prediction mode, the cost can be calculated based on the costs of the top-left and top-right templates.
[0311] As another example, when calculating the cost per mode, the cost of the corresponding prediction mode can be calculated based on the directionality of the prediction mode or the prediction mode number and template weights. For instance, the prediction mode number can be divided into J intervals, and the cost of the prediction mode can be calculated by weighting the costs of the top-left template, top-right template, top-left template, and bottom-left template according to each interval.
[0312] Meanwhile, as in the embodiment described above, by calculating the cost of the prediction mode based on the directionality of the prediction mode, more efficient computation can be enabled and intra-prediction performance can be improved.
[0313] Figures 19a and 19b show examples of group classification with different template configurations.
[0314] Referring to FIGS. 19a and 19b, groups by mode can be derived into m groups by comparing the costs of the top-left template, the top-right template, the top-left template, and the bottom-left template. Here, the groups may include at least one of the top group, the left group, and the non-directional group. For example, the groups may be derived based on the ratio or the magnitude of the difference between the sum of the costs of the top-left template and the top-right template and the sum of the costs of the top-left template and the bottom-left template.
[0315] Meanwhile, in the case of the non-directional prediction mode, groups can be implicitly derived as non-directional, or groups can be derived through a comparison of the costs of the top-left template and top-right template, and the costs of the top-left template and bottom-left template.
[0316] Specifically, FIG. 19a illustrates an example in which a prediction mode has a horizontal directionality and determines whether a specific condition is satisfied based on the ratio of the cost of the left template and the cost of the top template, or the magnitude of the difference, so that the group of the prediction mode is derived as the top group. Meanwhile, the specific condition may include the conditions in the following table.
[0317]
[0318] Here, the cost of the top-left template is cost AL and the cost of the top right template is cost AR and, the cost of the top-left template LA and the cost of the bottom-left template is cost LB And, is any positive real number for group classification. Any positive real number for group classification (parameter) For example, it can be predetermined, or, for another example, it can be changed depending on the aspect ratio of the block to be predicted or / and the size of the block to be predicted. Any positive real number (parameter) for the above group classification is the aforementioned group classification criteria (parameters) It could be the same as, or it could be different.
[0319] Specifically, FIG. 19b illustrates an example in which a prediction mode has a vertical orientation and determines whether a specific condition is satisfied based on the ratio of the cost of the left template and the cost of the top template, or the magnitude of the difference, so that the group of the prediction mode is derived as the left group. Meanwhile, the specific condition may include the conditions in the following table.
[0320]
[0321] Here, the cost of the top-left template is cost AL and the cost of the top right template is cost AR and, the cost of the top-left template LA and the cost of the bottom-left template is cost LB And, is any positive real number for group classification. Any positive real number for group classification (parameter) For example, it can be predetermined, or, for another example, it can be changed depending on the aspect ratio of the block to be predicted or / and the size of the block to be predicted. Any positive real number (parameter) for the above group classification is the aforementioned group classification criteria (parameters) It could be the same as, or it could be different.
[0322] Figures 20a and 20b show examples of group classification correction with different template configurations.
[0323] Referring to FIG. 20a and FIG. 20b, when determining a group by mode in the group by mode determination unit, the group of the prediction mode can be derived based on the directionality of the prediction mode. Alternatively, the group of the prediction mode can be changed.
[0324] For example, if a prediction mode has a horizontal directionality or is close to a horizontal prediction mode, the group corresponding to that prediction mode may be derived as a left group or a non-directional group. In this case, the top group may not be considered.
[0325] For example, if the prediction mode has a horizontal direction or is close to a horizontal prediction mode, and the sum of the costs of the top-left and top-right templates is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode may be derived as, for example, an upper group or a non-directional group based on a comparison of the costs of the top-left template and the bottom-left template, where the sum of the costs is calculated to be high. Alternatively, as another example, the group may be changed to a non-directional group.
[0326] For example, if the prediction mode has a horizontal directionality or is close to a horizontal prediction mode, and the ratio or difference between the cost of the top template and the cost of the left template is below a threshold, the group of the prediction mode may be derived as either an upper group or a non-directional group based on the ratio or difference between the cost of the top-left template and the cost of the bottom-left template. Meanwhile, the threshold may be any positive real number. Here, the cost of the left template or the cost of the top template may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively.
[0327] For example, if a prediction mode has a vertical orientation or is close to a vertical prediction mode, the group corresponding to that prediction mode may be derived as an upper group or a non-directional group. In this case, the left group may not be considered.
[0328] For example, if the prediction mode has a vertical orientation or is close to a vertical prediction mode, and the sum of the costs of the top-left and bottom-left templates is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode may be derived as a left group or a non-directional group, for example, based on a comparison of the costs of the top-left template and the top-right template, where the sum of the costs is calculated to be high. Alternatively, as another example, the group may be changed to a non-directional group.
[0329] For example, if the prediction mode has a vertical directionality or is close to a vertical prediction mode, and the ratio or difference between the cost of the top template and the cost of the left template is below a threshold, the group of the prediction mode may be derived as either a left group or a non-directional group based on the ratio or difference between the cost of the top-left template and the cost of the top-right template. Meanwhile, the threshold may be any positive real number. Here, the cost of the left template or the cost of the top template may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively.
[0330] For example, if the difference between the prediction mode number of a prediction mode and the prediction mode number of a horizontal prediction mode is K or less, the prediction mode may be determined to have a horizontal direction or to be close to a horizontal prediction mode. As another example, if the difference between the prediction mode number of a prediction mode and the prediction mode number of a vertical prediction mode is K or less, the prediction mode may be determined to have a vertical direction or to be close to a vertical prediction mode. K may represent the absolute value of the difference between the prediction mode number of the corresponding prediction mode and the prediction mode number of the horizontal or vertical prediction mode.
[0331] Here, the prediction mode number range K of the vertical prediction mode or the prediction mode close to the horizontal prediction mode may be predetermined, for example, or, for another example, changed according to the aspect ratio of the block to be predicted or / and the size of the block to be predicted. For example, when determining the range of the prediction mode close to the horizontal prediction mode, the range K may become smaller or larger as the width of the block becomes greater than the height. For another example, when determining the range of the prediction mode close to the vertical prediction mode, the range K may become smaller or larger as the height of the block becomes greater than the width.
[0332] Specifically, FIG. 20a illustrates an example where, when the prediction mode has horizontal directionality, an upper group or a non-directional group is derived based on the cost of the upper template and the cost of the left template. For example, even if the prediction mode has horizontal directionality and the sum of the costs of the upper-left template and the upper-right template is low, resulting in the upper group according to the group determination method described above, the group can be maintained as the upper group or changed to the non-directional group by determining whether a specific condition is satisfied based on a comparison of the costs of the upper-left template and the lower-left template, where the sum of the costs is calculated to be higher. In other words, when the prediction mode has horizontal directionality, one of the upper group or the non-directional group can be derived based on a comparison of the cost of the upper-left template and the cost of the lower-left template. In this case, if the cost of the lower-left template is greater than a certain ratio or difference than the cost of the upper-left template (i.e., equal or greater), the prediction mode can be derived as the upper group, and in other cases, the prediction mode can be derived as the non-directional group.
[0333] Meanwhile, the above specific conditions may include the conditions in the following table.
[0334]
[0335] Here, the cost of the top-left template LA and the cost of the bottom-left template is cost LB It is. Meanwhile, is any positive real number for determining the group to be changed, and, for example, can be predetermined, or, for other example, can be changed according to the aspect ratio of the block to be predicted or / and the size of the block to be predicted.
[0336] Specifically, FIG. 20b illustrates an example where, when the prediction mode has vertical directionality, a left group or a non-directional group is derived based on the cost of the top template and the cost of the left template. For example, even if the prediction mode has vertical directionality and the sum of the costs of the top-left and bottom-left templates is low, resulting in a left group according to the group determination method described above, the group can be maintained as a left group or changed to a non-directional group by determining whether a specific condition is satisfied based on a comparison of the costs of the top-left template and the top-right template, where the sum of the costs is calculated to be higher. In other words, when the prediction mode has vertical directionality, one of the left group or a non-directional group can be derived based on a comparison of the cost of the top-left template and the cost of the top-right template. In this case, if the cost of the top-right template is greater than a certain ratio or difference than the cost of the top-left template (i.e., equal or greater), the prediction mode can be derived as a left group, and in other cases, the prediction mode can be derived as a non-directional group.
[0337] Meanwhile, the above specific conditions may include the conditions in the following table.
[0338]
[0339] Here, the cost of the top-left template AL and the cost of the top right template is cost AR It is. Meanwhile, is any positive real number for determining the group to be changed, and, for example, can be predetermined, or, for other example, can be changed according to the aspect ratio of the block to be predicted or / and the size of the block to be predicted.
[0340] Meanwhile, the accuracy of the final prediction result can be increased by deriving groups or correcting group classification using the directionality of the prediction mode, and the prediction quality can be improved by mitigating the discrepancy between the directionality of the prediction mode and the directionality of group classification.
[0341] Figure 21 shows an example of a method for generating a TIMD prediction block with changed conditions.
[0342] Referring to FIG. 21, the prediction unit of the coding device may include a prediction mode determination unit (2100), a prediction block generation unit by mode (2110), a first weighted sum execution unit by group (2120), and a pixel location-based second weighted sum execution unit (2130). The prediction mode determination unit (2100), the prediction block generation unit by mode (2110), the first weighted sum execution unit by group (2120), and the pixel location-based second weighted sum execution unit (2130) may perform the same or similar operations as the prediction mode determination unit, the prediction block generation unit by mode, the first weighted sum execution unit by group, and the pixel location-based second weighted sum execution unit described in FIG. 10.
[0343] Meanwhile, when the pixel location-based second weighted sum execution unit (2130) determines whether to perform a second weighted sum, it may determine whether there are prediction blocks of different groups. In this case, if only the prediction blocks of the upper or left group exist as a result of the group-specific first weighted sum execution unit (2120), the phenomenon in which the result of the pixel location-based second weighted sum execution becomes identical to the result of the group-specific first weighted sum execution can be prevented, thereby preventing the execution of unnecessary computation procedures.
[0344] For example, group-unit flags for the left group, top group, and non-directed group can be configured (e.g., groupLflag, groupAflag, groupNflag), and if the sum of the three flags is 2 or more, it is determined that different group prediction blocks exist, and the pixel location-based second weighted sum can be performed. Alternatively, for example, a different group flag (diffgroupflag) can be configured as an internal parameter, and if different group prediction blocks exist, the different group flag can be set to 1, and if the different group flag is 1, the pixel location-based second weighted sum can be performed. As another example, based on the number of groups, if the number of groups is 2 or more, the pixel location-based second weighted sum can be performed.
[0345] According to one embodiment of the method for generating TIMD prediction blocks described above, a pixel location-based second weighted sum is performed when an upper group or a left group exists. In this case, a pixel location-based second weighted sum may also be performed when only an upper group exists or only a left group exists. That is, the result before and after the pixel location-based second weighted sum is performed becomes identical, and there may be a problem that unnecessary computational procedures are performed. However, according to the present embodiment, fusion or weighted sum of unnecessary intra prediction blocks is prevented, thereby reducing coding complexity and increasing computational efficiency.
[0346] Meanwhile, as described above, the encoding device can derive a residual signal based on a prediction signal (including a prediction block). The conversion unit of the encoding device can generate conversion coefficients by performing a conversion on the residual signal of the current block, and the inverse conversion unit of the encoding device and the decoding device can restore the residual signal by performing an inverse conversion on the conversion coefficients.
[0347] In the present disclosure, inverse transformation may be referred to as transformation depending on the case. For example, an inverse transformation kernel may be called a transformation kernel. For example, a transformation kernel may be a transformation kernel with respect to the transformation procedure of an encoder, and an inverse transformation kernel with respect to the inverse transformation procedure of an encoder / decoder. A second (inverse)transform kernel may include a non-separable (inverse)transform kernel. A first (inverse)transform kernel may include a separable (inverse)transform kernel.
[0348] Figure 22 illustrates first-order and second-order transformation techniques as examples.
[0349] The conversion unit may correspond to the conversion unit within the encoding device described above, and the inverse conversion unit may correspond to the inverse conversion unit within the encoding device described above or the inverse conversion unit within the decoding device.
[0350] The transformation unit can derive (primary) transformation coefficients by performing a primary transformation based on residual samples (residual sample array) within the residual block (S2200). This primary transformation can be performed based on a DCT-based transformation kernel, a DST-based transformation kernel and / or an identity transformation kernel, etc. The primary transformation can be based on MTS (multiple transform selection).
[0351] The transformation unit can derive modified (second) transformation coefficients by performing a second transformation based on the (first) transformation coefficients (S2210). The first transformation is a transformation from the spatial domain to the frequency domain, and the second transformation can represent a transformation into a more compressed expression using the correlation existing between the (first) transformation coefficients.
[0352] For example, the second-order transformation may include a non-separable transform. The second-order transformation may be referred to as a low-frequency non-separable transform (LFNST). The second-order transformation may represent a transformation that generates modified transformation coefficients (or second-order transformation coefficients) for a residual signal by performing a second transformation on the (first-order) transformation coefficients derived through the first-order transformation based on a non-separable transform matrix. Here, the transformation may be applied at once to the (first-order) transformation coefficients based on the non-separable transform matrix, without applying the vertical and horizontal transformations separately (or independently). The (non-separable) transformation matrix may be derived by offline training based on a specific dataset. For example, when a second-order (inverse)transform is applied, the transformation kernel for the first-order (inverse)transform may be restricted. For example, in this case, only a DCT-based kernel or a DST-based kernel can be used as the transformation kernel for the first (inverse) transformation.
[0353] That is, the above-mentioned inseparable second transformation may represent a transformation method that does not separate the vertical and horizontal components of the above-mentioned (first) transformation coefficients, for example, rearranges two-dimensional signals (transformation coefficients) into one-dimensional signals through a specific defined direction, and then generates modified transformation coefficients (or second transformation coefficients) based on the above-mentioned inseparable transformation matrix.
[0354] Quantized transformation coefficients can be derived through a quantization procedure for the above second transformation coefficients. Inversely quantized transformation coefficients can be derived through an inverse quantization procedure for the quantized transformation coefficients.
[0355] Additionally, the inverse transform unit may perform a series of procedures in the reverse order of the procedures performed in the transform unit described above. The inverse transform unit receives (inversely quantized) transform coefficients, performs a second (inverse) transform to derive (first) transform coefficients (S2220), and performs a first (inverse) transform on the (first) transform coefficients to obtain residual blocks (residual samples) (S2230). Here, the first transform coefficients may be referred to as modified transform coefficients on the side of the inverse transform unit. As described above, the encoding device and / or decoding device may generate a restored block based on the residual block and the predicted block, and may generate a restored picture based thereon.
[0356] In some cases, a specific transformation (including an inverse transformation) may be performed instead of the above first (inverse) transformation and the above second (inverse) transformation. The specific transformation may be based on a non-separable transformation. In this case, the specific transformation may be called an integrated transformation, NSPT (non-separable primary transform) or NSUT (non-separable unified transform). Whether the specific transformation is applied may be determined based on the block size.
[0357] FIG. 23 illustrates an exemplary residual processing procedure including a second transformation / inverse transformation.
[0358] The first-order transformed transformation coefficients can be spatially arranged in the target block (transformation block). In this case, a second-order transformation can be applied to the transformation coefficients in the upper-left low-frequency region to further compress the signal. In this case, the first M transformation coefficients (or sub-blocks) are derived according to the scan order and can be used as inputs for the second-order transformation. M can be variably set based on the block size. The number of (valid) transformation coefficients for the output of the second-order transformation may be smaller than the number of (valid) transformation coefficients for the input of the second-order transformation. A transformation matrix (transform kernel or transformation kernel matrix) may be used for the second-order transformation. For example, a transformation set is determined based on the prediction mode of the target block, and a transformation matrix (transform kernel or transformation kernel matrix) within the transformation set may be selected. Index information may be signaled for the selection of the transformation matrix.
[0359] For the inverse transformation, the transposed matrix of the transformation matrix may be used, or the input may be multiplied in the direction transposed based on the transformation matrix. When an inverse quadratic transformation is performed as an inverse transformation process performed in a decoding device, the input coefficient data to which the inverse quadratic transformation is to be applied is composed of a one-dimensional vector according to a predetermined array order, and a modified coefficient vector obtained by multiplying the one-dimensional vector by the corresponding transformation matrix may be arranged in a two-dimensional block according to a predetermined array order.
[0360] Meanwhile, according to one embodiment of the present disclosure, with respect to a non-regular intra block, a transformation kernel associated with the block may be selected based on reference samples and / or prediction samples. In other words, an optimal transformation kernel associated with the current transformation block may be derived based on directional information of already restored reference samples or prediction samples.
[0361] Regarding non-regular intra blocks, which are blocks coded (predicted or restored) based on prediction types such as DC, Planar, or directional modes among intra prediction methods / types, the transformation kernel can be derived based on the directional information of the restoration reference sample. For example, regarding regular intra blocks, the intra prediction mode of the block can be signaled based on a (primary / secondary) MPM (candidate) list, or the intra prediction mode of the block can be signaled through remaining mode information indicating one of the remaining modes excluding the (primary / secondary) MPM candidates. In addition, for example, a non-regular intra block may represent a block coded (predicted or restored) based on prediction types such as DIMD (decoder side intra mode derivation), TIMD (fusion for template-based intra mode derivation or template-based intra mode derivation), MIP (matrix-based intra sample prediction), intraTMP (intra template matching prediction), OBIC (occurrence-based intra coding), TMRL (template-based multiple reference line intra prediction), SGPM (spatial Geometric partitioning mode), EIP (extrapolation filter-based intra prediction) and / or NN_intra (neural network-based intra prediction).In some cases, the embodiments for non-regular intra blocks may also be applied to blocks related to inter-prediction.
[0362] Figure 24 illustrates an exemplary HoG (Histogram of Gradient) related to directionality.
[0363] As shown in Fig. 24, a Histogram of Gradient (HoG) related to directionality can be derived based on a restored reference sample and / or a predicted sample.
[0364] For example, directional information can be derived based on a restoration reference sample and / or a prediction sample, and a transformation kernel can be derived based on said directional information.
[0365] Based on the above HoG, the directional mode M with the greatest intensity (intensity of directionality) 1st can be selected. Subsequently, the transformation block size or the directional mode M above can be selected. 1st Based on at least one of the above, a transformation kernel can be determined by referring to a Look Up Table (LUT) in which a set of transformation kernels is defined.
[0366] Additionally, for example, a separate LUT can be derived for each intra-prediction mode, and the top two directional modes (M) with the highest directional strength based on HoG 1st ,M 2nd ) can be selected, and the (number) difference (IPM) of the two directional modes above diff = |M 1st - M 2nd Based on |), groups according to the difference between the two directional modes can be divided into N groups. For example, as shown in the following table, N can be 3.
[0367]
[0368] For example, as shown in the following table, the above group (IPM diff,idx), transform block size, directional mode M 1st A transformation kernel can be determined from a LUT associated with an intra prediction mode based on at least one of the following.
[0369]
[0370] FIG. 25 shows an example of a decoding device that performs a decoding procedure to which the present embodiment can be applied. The same description of the above configuration may be omitted.
[0371] Referring to FIG. 25, the decoding device may include a prediction mode determination unit (2500), a prediction execution unit (2510), a first / second inverse transform kernel determination unit (2520), a transform coefficient entropy decoding unit (2530), a transform coefficient inverse quantization unit (2540), an inverse transform execution unit (2550), and a restoration execution unit (2560). For example, as shown in the block diagram of FIG. 25, a prediction mode may be determined for the current block. Additionally, a restoration signal may be generated by adding a prediction signal predicted by the prediction mode and a residual signal for which (inverse) transform has been performed.
[0372] In this document, the order of each process in the block diagram may be changed according to the embodiment, and some processes may be omitted.
[0373] In this document, inverse transform and inverse quantization correspond to the transform and quantization performed in the encoding device, and may be expressed as (inverse) transform and (inverse) quantization or transform and quantization. For example, an inverse transform kernel may be referred to as a transform kernel. Alternatively, for example, a transform kernel may be a transform kernel associated with a transform procedure performed in an encoding procedure, or an inverse transform kernel associated with an inverse transform procedure performed in an encoding / decoding procedure.
[0374] The prediction mode determination unit (2500) and the prediction execution unit (2510) may be included in the prediction unit of the decoding device described above. Additionally, the first / second inverse transform kernel determination unit (2520) and the inverse transform execution unit (2550) may be included in the inverse transform unit of the decoding device described above. Additionally, the transform coefficient inverse quantization unit (2540) may be included in the inverse quantization unit of the decoding device described above. Additionally, the transform coefficient entropy decoding unit (2530) may be included in the entropy decoding unit of the decoding device described above. Additionally, the restoration execution unit (2560) may be included in the addition unit of the decoding device described above.
[0375] In the prediction mode determination unit (2500), a prediction method / type such as intra prediction, inter prediction, IBC (Intra block copy) prediction, palette prediction, and intra-inter-frame weighted sum prediction mode of the current block can be determined, and a prediction mode to be performed on the prediction of the current block can be determined based on the determined prediction method / type.
[0376] In the prediction execution unit (2510), a prediction for the current block can be performed based on the prediction method / type and prediction mode determined by the prediction mode determination unit (2500).
[0377] In the inverse quantization unit (2540) of the conversion coefficient, inverse quantization can be performed on the conversion coefficient level value of the current conversion block size that has been entropy-decoded in the conversion coefficient entropy decoding unit (2530). In some cases, the (inverse) quantization process may be omitted.
[0378] In the first / second inverse transform kernel determination unit (2520), a first or second (inverse) transform kernel may be determined. For example, the second (inverse) transform kernel may include a non-separable (inverse) transform kernel, and the first (inverse) transform kernel may include a separable (inverse) transform kernel. Additionally, for example, the second (inverse) transform may be applied to the upper-left region (low-frequency transform coefficient region) of the transform block.
[0379] In the inverse transformation execution unit (2550), an inverse transformation for the current transformation block can be performed based on the (inverse) transformation kernel. In some cases, the (inverse) transformation process may be omitted.
[0380] FIG. 26 exemplarily illustrates the procedure for determining the first / second inverse transform kernel in a coding device.
[0381] Referring to FIG. 26, the first / second (inverse) transformation kernel of the current block in the coding device can be determined by parsing or deriving a flag or index.
[0382] For example, if inseparable inverse transformation is not performed, whether to perform Multiple Transform Selection (MTS) can be determined. The determination of whether to perform MTS may be based on at least one of the following: an enabled flag or index-related information at a higher level, flag or index-related information parsed to determine the inseparable inverse transformation kernel, the existence of transformation factors, or transform skip flag or index-related information. For example, the higher level may be a high-level syntax (HLS) such as SPS or PPS.
[0383] For example, if the parsed enabled flag is 1, the parsed index to determine the non-separable inverse transform kernel is 0, there are no transform coefficients, and the parsed transform skip flag is 0, MTS can be performed.
[0384] For example, if MTS is not performed, the (inverse)transform kernels in the vertical / horizontal directions may be explicitly or implicitly determined based on whether intra-prediction is performed on the current block or the size of the current transform block. For example, if MTS is not performed, the transform kernels for all transform blocks may be determined as DCT-2 kernels.
[0385] For example, when MTS is performed, the transformation kernel can be determined based on whether it is intra-frame prediction or inter-frame prediction. For instance, by parsing flag or index-related information indicating whether it is intra-frame prediction or inter-frame prediction, the transformation kernel can be determined through subsequent transformation kernel determination procedures.
[0386] Meanwhile, whether to perform the intra-frame prediction Implicit MTS inverse transformation kernel determination unit (2660) can be determined based on at least one of the enabled flag parsed from the upper level, whether the current block is a luminance component, whether the current block is an intra prediction, and whether an ISP (Intra Sub-Partition) is applied. For example, if the parsed enabled flag is 1, the current block is a luminance component, the current block is predicted as an intra prediction type, and the ISP (Intra Sub-Partition) mode is not applied, the intra-frame prediction Implicit MTS inverse transformation kernel determination unit (2660) can be performed.
[0387] When the in-frame prediction Implicit MTS inverse transformation kernel determination unit (2660) is performed, the primary (inverse) transformation kernel determination method may be different based on whether the current block is a regular intra block predicted based on a directional mode, planar mode, or DC mode, or a non-regular intra block predicted based on the aforementioned DIMD, TIMD, SGPM, MIP, TMRL, NN_intra, etc.
[0388] FIG. 27 illustrates an example of a method for determining a first / second inverse transform kernel for a non-regular intra block.
[0389] Referring to FIG. 27, the Regular intra-block determination unit (2700) can parse flag or index-related information regarding the intra-block mode of the current block. Additionally, the Regular intra-block execution unit (2710) can select a first-order inverse transformation kernel determination method based on the flag or index-related information parsed by the Regular intra-block determination unit (2700). Additionally, the directional mode parsing unit (2720) can parse the directional mode, and the directional mode and strength derivation unit (2730) can derive the directional mode and strength. Additionally, the inverse transformation kernel determination unit (2740) can determine the first-order inverse transformation kernel based on at least one of the transformation block size and the parsed directional mode.
[0390] For example, if the current block is a regular intra block, a first inverse transformation kernel can be determined based on at least one of the transformation block size and the directional mode (or non-directional prediction mode). Specifically, the inverse transformation kernel of the current block can be determined based on the intra-directional or non-directional prediction mode or the current transformation block size, which is determined by parsing information related to the directional mode index through the directional mode parsing unit (2720).
[0391] For example, if the current block is a non-regular intra block, a first-order inverse transformation kernel can be determined based on at least one of the following: an intra-frame prediction mode (DIMD, TIMD, SGPM, MIP, TMRL, NN_intra, etc.), a transformation block size, a directional mode, the difference between two directional modes, and the distribution characteristics of the HoG (Histogram of Gradient). Specifically, the distribution characteristics of the HoG may represent N directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described in the explanation of FIG. 28 below.
[0392] Figure 28 illustrates an exemplary method for deriving a HoG (Histogram of Gradient) related to directionality.
[0393] Referring to FIG. 28, the HoG associated with the directionality can be derived based on the restored reference sample or prediction signal. For example, the size and location of the restored reference sample area and the prediction sample area can be determined based on the prediction mode of the current block, the block size, or the aspect ratio of the block, and the size of the area may not be determined to be at the same location as the rightmost or bottommost pixel line of the current block as shown in FIG. 28. For example, the bottom of the reference area may be located below or above the bottom pixel line of the current block, and the right side of the reference area may be located to the right or left of the right side of the block. Alternatively, the reference area may be located in a form that is not immediately adjacent to the current block.
[0394] Meanwhile, the directional mode and intensity derivation unit (2730) of FIG. 27 derives a HoG (Histogram of Gradient) related to directionality, and the top N directional modes (M) with high directional intensity. 1st , M 2nd , ... , M N ) and the top N directional intensities with large directional strengths (S(M 1st ), S(M 2nd ), ... , S(M N )) can be derived. In this case, N can be a positive integer.
[0395] In addition, the inverse transformation kernel determination unit (2740) of FIG. 27 includes an in-frame prediction mode, a transformation block size, and the M 1st , above M 1st and the above M 2nd The difference of (|M 1st - M 2nd A first-order inverse transform kernel can be determined based on at least one of the distribution characteristics of the above HoG. Specifically, a first-order inverse transform kernel can be determined based on the distribution characteristics of the HoG related to N directional intensities.
[0396] FIGS. 29 and 30 exemplarily show how a first-order inverse transform kernel is determined when the intensity of the directionality is evenly distributed.
[0397] Referring to FIG. 29, with respect to the HoG distribution characteristics, the directional mode M with the greatest directional intensity 1st The intensity of S(M 1st ) is smaller than the threshold TH1, and the above S(M 1st ) and the second largest directional mode M 2nd The intensity of S(M 2nd Difference from ) (S diff = S(M 1st ) - S(M 2nd If )) is smaller than the threshold TH2, it can be determined that the intensity of the directionality is evenly distributed. For example, the two intensity values of the directionality may represent the histogram values of the top two directional modes with the largest histogram sizes among the directional modes derived through mapping to the HoG directional modes. Additionally, whether the intensity of the directionality is evenly distributed can be determined based on two or more intensity values of directionality.
[0398] Referring to FIG. 30, the average of N directional intensities (S avg ) and the above S(M 1st If the difference of ) is smaller than the threshold TH3, it can be determined that the intensity of the directionality is evenly distributed. For example, the N directional intensities may represent the intensity of the top N directional modes with the largest directional intensity in the above HoG.
[0399] As described above, when the intensity of the directions is evenly distributed, for example, because all directions have similar influence, a specific inverse transform kernel can be determined for the horizontal and vertical directions. For example, the specific inverse transform kernel may be a DCT-2 kernel.
[0400] Alternatively, for example, the directional mode of the current block can be selected or considered as DC mode or Planar mode, and the inverse transform kernel of the transform block can be determined according to the Regular intra block inverse transform kernel determination method described above.
[0401] Alternatively, for example, a kernel set (vertical, horizontal) related to the case where the intensity of directionality is evenly distributed is separately derived or predefined, such that the in-frame prediction mode of the current block, the size of the transformation block, and the above M 1st , M 1st and M 2nd The difference of (|M 1st - M 2nd The inverse transformation kernel can be determined based on at least one of |).
[0402] FIGS. 31 and 32 exemplarily illustrate how a first-order inverse transform kernel is determined when the strength of one directionality is dominant.
[0403] Referring to FIG. 31, with respect to the HoG distribution characteristics, the directional mode M with the greatest directional intensity 1st The intensity of S(M 1st ) and the second largest directional mode M 2nd The intensity of S(M 2nd Difference from ) (S diff = S(M 1st ) - S(M 2nd If )) is greater than the threshold TH4, the strength of one direction, i.e., the above S(M 1st It can be judged that ) is dominant.
[0404] Referring to FIG. 32, the average of N directional intensities (S avg ) and the above S(M 1st If the difference of ) is greater than the threshold TH5, the strength of one direction, i.e., the above S(M 1stIt can be determined that ) is dominant. For example, N directional intensities may represent the intensities of the top N directional modes with large directional intensities in the above HoG.
[0405] As described above, in the case where the strength of one directionality is dominant, for example, among the groups that can be divided into N based on the difference between the two directionality modes described above, the group with a small difference in directionality modes (0 < |M 1st - M 2nd By mapping to | <= m, where m can be a positive integer), the (inverse)transformation kernel can be determined based on this. For example, the above group (IPM diff,idx The index of ) can be determined to be 0, and a transformation kernel can be determined based on the index and LUT.
[0406] Or, for example, the directional mode of the current block is M 1st By selecting or considering it as such, the inverse transformation kernel of the transformation block can be determined according to the Regular intra block (inverse) transformation kernel determination method described above.
[0407] Alternatively, for example, a kernel set (vertical, horizontal) associated with a case where one directionality is dominant is separately derived or predefined, such that the in-frame prediction mode of the current block, the size of the transformation block, and the above M 1st , M 1st and M 2nd The difference of (|M 1st - M 2nd The inverse transformation kernel can be determined based on at least one of |).
[0408] Meanwhile, for example, in cases where the intensity of the directionality is not evenly distributed and the intensity of one directionality is not dominant, the in-screen prediction mode, the size of the transformation block, and the above M 1st , M 1st and M 2nd The difference of (|M 1st - M 2ndThe inverse transformation kernel can be determined based on at least one of |).
[0409] Meanwhile, whether the intra-frame prediction Explicit MTS inverse transformation kernel determination unit (2640) of FIG. 26 is executed can be determined based on an enabled flag parsed from a higher level, whether the current block is a luminance component, etc. For example, if the parsed enabled flag is 1 and the current block is a luminance component, the intra-frame prediction Explicit MTS inverse transformation kernel determination unit (2640) can be executed.
[0410] When the in-frame prediction Explicit MTS inverse transformation kernel determination unit (2640) is performed, the primary (inverse) transformation kernel determination method may be different based on whether the current block is a regular intra block predicted based on a directional mode, planar mode, or DC mode, or a non-regular intra block predicted based on the aforementioned DIMD, TIMD, SGPM, MIP, TMRL, NN_intra, etc.
[0411] FIG. 33 shows an example of a method for determining a first / second inverse transform kernel for a non-regular intra block.
[0412] Referring to FIG. 33, the Regular intra-block determination unit (3300) can parse flag or index-related information regarding the intra-block mode of the current block. Additionally, the Regular intra-block execution unit (3310) can select a first-order inverse transformation kernel determination method based on the flag or index-related information parsed by the Regular intra-block determination unit (3300). Additionally, the directional mode parsing unit (3320) can parse the directional mode, and the directional mode and intensity derivation unit (3330) can derive the directional mode and intensity. Additionally, the mts_idx parsing unit (3340) can parse index information related to transformation kernel determination. For example, mts_idx can be greater than or equal to 0 and less than k. In this case, k can be a positive integer. Additionally, the inverse transformation kernel determination unit (3350) can determine a primary inverse transformation kernel based on at least one of the transformation block size and the directional mode. For example, information related to a transformation kernel set table index can be derived based on at least one of the transformation block size and the directional mode, and a transformation kernel set table can be derived based on the index. A transformation kernel set can be derived based on the transformation kernel set table and information related to the parsed mts_idx, and a primary inverse transformation kernel can be determined based on the transformation kernel set.
[0413] For example, if the current block is a regular intra block, mts_idx can be derived based on at least one of the transformation block size and the directional mode (or non-directional prediction mode), and a first-order inverse transformation kernel can be determined based on the mts_idx. Specifically, the inverse transformation kernel of the current block can be determined based on the intra-directional or non-directional prediction mode or the current transformation block size, etc., which are determined by parsing information related to the directional mode index through the directional mode parsing unit (3320).
[0414] For example, if the current block is a non-regular intra block, as described above, mts_idx can be derived based on at least one of the following: an intra-frame prediction mode (DIMD, TIMD, SGPM, MIP, TMRL, NN_intra, etc.), a transformation block size, a directional mode, the difference between two directional modes, and a distribution characteristic of HoG (Histogram of Gradient), and a first-order inverse transformation kernel can be determined based on said mts_idx. Specifically, the distribution characteristic of said HoG may represent N-directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described above. Specifically, the distribution characteristic of said HoG may represent N-directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described above in Fig. 28.
[0415] Meanwhile, the directional mode and intensity derivation unit (3330) derives a HoG (Histogram of Gradient) related to directionality and the top N directional modes (M) with high intensity of directionality 1st , M 2nd , ... , M N ) and the top N directional intensities with large directional strengths (S(M 1st ), S(M 2nd ), ... , S(M N)) can be derived. In this case, N can be a positive integer.
[0416] Additionally, the mts_idx parsing unit (3340) can parse index information related to the conversion kernel determination. For example, mts_idx may be greater than or equal to 0 and less than k. In this case, k may be a positive integer.
[0417] Additionally, the inverse transform kernel determination unit (3350) can determine a first-order inverse transform kernel. For example, an in-frame prediction mode, a transform block size, and the M 1st , above M 1st and the above M 2nd The difference of (|M 1st - M 2nd mts_idx can be derived based on at least one of the distribution characteristics of the above HoG, and a first-order inverse transformation kernel can be determined based on the above mts_idx. Specifically, a first-order inverse transformation kernel can be determined based on at least one of the distribution characteristics of the HoG related to directional mode information or N directional intensities.
[0418] For example, a first-order inverse transform kernel can be determined based on directional mode information. Specifically, the intra-frame prediction mode, the transform block size, and the above M 1st , above M 1st and the above M 2nd The difference of (|M 1st - M 2nd |), mts_idx can be derived based on at least one of the distribution characteristics of the above HoG, and a first-order inverse transform kernel can be determined based on mts_idx. mts_idx may be greater than or equal to 0 and less than k. In this case, k may be a positive integer. Also, as described above, the difference between two directional modes (|M 1st - M 2nd p groups divided based on |) (0 <= IPM diff,idxOne of < p can be derived. In this case, p can be a positive integer.
[0419] In addition, for example, a first / second inverse transform kernel can be determined based on the distribution of multiple directional intensities.
[0420] Specifically, for example, the directional mode M with the greatest directional strength 1st The intensity of S(M 1st ) is smaller than the threshold TH1, and the above S(M 1st ) and the second largest directional mode M 2nd The intensity of S(M 2nd Difference from ) (S diff = S(M 1st ) - S(M 2nd If )) is smaller than the threshold TH2, it can be determined that the intensity of the directionality is evenly distributed. For example, the two intensity values of the directionality may represent the histogram values of the top two directional modes with the largest histogram sizes among the directional modes derived through mapping to the HoG directional modes. Additionally, whether the intensity of the directionality is evenly distributed can be determined based on two or more intensity values of directionality.
[0421] Also, for example, the average of N directional intensities (S avg ) and the above S(M 1st If the difference of ) is smaller than the threshold TH3, it can be determined that the intensity of the directionality is evenly distributed. For example, the N directional intensities may represent the intensity of the top N directional modes with the largest directional intensity in the above HoG.
[0422] As described above, when the intensity of the directionality is evenly distributed, for example, because all directional directions have similar influence, a specific (inverse)transformation kernel can be determined for the horizontal and vertical directions. For example, the specific (inverse)transformation kernel may be a DCT-2 kernel.
[0423] Alternatively, for example, the directional mode of the current block can be selected or considered as DC mode or Planar mode, and the inverse transform kernel of the transform block can be determined according to the Regular intra block inverse transform kernel determination method described above.
[0424] Alternatively, for example, a kernel set (vertical, horizontal) related to the case where the intensity of directionality is evenly distributed is separately derived or predefined, such that the in-frame prediction mode of the current block, the size of the transformation block, and the above M 1st , M 1st and M 2nd The difference of (|M 1st - M 2nd mts_idx can be derived based on at least one of |), and an inverse transformation kernel can be determined based on the said mts_idx.
[0425] Meanwhile, for example, the directional mode M with the greatest directional strength 1st The intensity of S(M 1st ) and the second largest directional mode M 2nd The intensity of S(M 2nd Difference from ) (S diff = S(M 1st ) - S(M 2nd If )) is greater than the threshold TH4, the strength of one direction, i.e., the above S(M 1st It can be judged that ) is dominant.
[0426] Or, for example, the average of N directional intensities (S avg ) and the above S(M 1st If the difference of ) is greater than the threshold TH5, the strength of one direction, i.e., the above S(M 1st It can be determined that ) is dominant. For example, N directional intensities may represent the intensities of the top N directional modes with large directional intensities in the above HoG.
[0427] As described above, in the case where the strength of one directionality is dominant, for example, among the groups that can be divided into N based on the difference between the two directionality modes described above, the group with a small difference in directionality modes (0 < |M 1st - M 2nd By mapping to | <= m, where m can be a positive integer), the (inverse)transformation kernel can be determined based on this. For example, the above group (IPM diff,idx The index of ) can be determined to be 0, and a transformation kernel can be determined based on the index and LUT.
[0428] Or, for example, the directional mode of the current block is M 1st By selecting or considering it as such, the inverse transformation kernel of the transformation block can be determined according to the Regular intra block (inverse) transformation kernel determination method described above.
[0429] Alternatively, for example, a kernel set (vertical, horizontal) associated with a case where one directionality is dominant is separately derived or predefined, such that the in-frame prediction mode of the current block, the size of the transformation block, and the above M 1st , M 1st and M 2nd The difference of (|M 1st - M 2nd mts_idx can be derived based on at least one of |), and an inverse transformation kernel can be determined based on the said mts_idx.
[0430] Meanwhile, when a non-separable (inverse)transformation is performed, an inseparable (inverse)transformation kernel may be determined in the non-separable inverse transformation kernel determination unit (2600) of FIG. 26.
[0431] For example, whether to perform the non-separable inverse transformation kernel determination unit (2600) can be determined based on the enabled flag or index-related information parsed at the upper level, the flag or index-related information parsed in relation to the kernel determination, etc. For example, if the parsed enabled flag is 1 and index-related information related to the kernel determination is parsed, the non-separable inverse transformation can be performed when the index related to the kernel determination is greater than 0.
[0432] For example, whether to perform a non-separable first-order inverse transform can be determined based on the size of the current transform block. If the current transform block size is sufficiently small, an NSPT related to the first-order inverse transform may be performed, otherwise, an LFNST related to the second-order inverse transform may be performed.
[0433] Specifically, for example, regarding the size of the current transformation block, if the width is smaller than the threshold th1 and the height is greater than or equal to the threshold th2, NSPT can be performed.
[0434] In addition, for example, with respect to the size of the current transformation block, if the width is greater than or equal to the threshold th1 and the height is greater than or equal to the threshold th2, LFNST can be performed.
[0435] When NSPT / LFNST is performed, the NSPT / LFNST kernel may be determined based on at least one of the NSPT / LFNST kernel size, NSPT / LFNST kernel set, NSPT / LFNST kernel flags, and index-related information.
[0436] Meanwhile, when the non-separable inverse transformation kernel determination unit (2600) is performed, the primary (inverse) transformation kernel determination method may be different based on whether the current block is a regular intra block predicted based on a directional mode, planar mode, or DC mode, or a non-regular intra block predicted based on the aforementioned DIMD, TIMD, SGPM, MIP, TMRL, NN_intra, etc.
[0437] FIG. 34 illustrates an example of a method for determining a first / second inverse transform kernel for a non-regular intra block.
[0438] Referring to FIG. 34, the Regular intra-block determination unit (3400) can parse information related to flags or indices regarding the intra-block mode of the current block. Additionally, the Regular intra-block execution unit (3410) can select a method for determining first / second inverse transformation kernels based on the flags or indices related to the information parsed by the Regular intra-block determination unit (3400). Additionally, the directional mode and strength derivation unit (3420) can derive a directional mode and strength, and the kernel set derivation unit (3430) can derive an NSPT / LFNST kernel set based on the directional mode. Additionally, the kernel parsing unit (3440) can parse information related to NSPT / LFNST kernel flags or indices regarding which NSPT / LFNST kernel to use from the selected NSPT / LFNST kernel set. Additionally, the kernel determination unit (3450) can determine the NSPT / LFNST kernel based on at least one of the NSPT / LFNST kernel size, NSPT / LFNST kernel set, NSPT / LFNST kernel flag, or index-related information.
[0439] For example, if the current block is a regular intra-block, the NSPT / LFNST kernel can be determined based on at least one of the NSPT / LFNST kernel size, NSPT / LFNST kernel set, NSPT / LFNST kernel flag, or index-related information.
[0440] For example, if the current block is a non-regular intra block, as described above, the intra-frame prediction mode, transform block size, M 1st , M 1st and M 2nd The difference of (|M 1st - M 2ndThe NSPT / LFNST kernel can be determined based on at least one of the distribution characteristics of the HoG, the NSPT / LFNST kernel size, the NSPT / LFNST kernel set, the NSPT / LFNST kernel flag, or index-related information. Specifically, the distribution characteristics of the HoG may represent N directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described above. Specifically, the distribution characteristics of the HoG may represent N directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described above in Fig. 28.
[0441] Meanwhile, the directional mode and intensity derivation unit (3420) derives a HoG (Histogram of Gradient) related to directionality, and the top N directional modes (M) with high directional intensity. 1st , M 2nd , ..., M N ) and the top N directional intensities with large directional strengths (S(M 1st ), S(M 2nd ), ..., S(M N )) can be derived. In this case, N can be a positive integer.
[0442] Additionally, the kernel set induction unit (3430) can induce an NSPT / LFNST kernel set based on a directional mode.
[0443] Additionally, the kernel parsing unit (3440) can parse NSPT / LFNST kernel flag or index-related information regarding which NSPT / LFNST kernel to use from the selected NSPT / LFNST kernel set.
[0444] In addition, the kernel decision unit (3450) includes an intra-frame prediction mode, a conversion block size, and the M 1st , above M 1st and the above M 2nd The difference of (|M 1st - M 2ndThe NSPT / LFNST kernel can be determined based on at least one of the distribution characteristics of the HoG, the NSPT / LFNST kernel size, the NSPT / LFNST kernel set, the NSPT / LFNST kernel flag, or index-related information. Specifically, the distribution characteristics of the HoG may represent N directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described above. Specifically, the distribution characteristics of the HoG may represent N directional intensity distribution characteristics derived based on the restored reference sample or prediction signal, as described above in Fig. 28.
[0445] For example, a first / second inverse transform kernel can be determined based on directional mode information. Specifically, the intra-frame prediction mode, the transform block size, and the above M 1st , above M 1st and the above M 2nd The difference of (|M 1st - M 2nd |), NSPT / LFNST flag or index-related information can be derived based on at least one of the distribution characteristics of the above HoG, and first / second inverse transform kernels can be determined based on the NSPT / LFNST flag or index-related information. In addition, as described above, the difference between the two directional modes (|M 1st - M 2nd p groups divided based on |) (0 <= IPM diff,idx One of < p can be derived. In this case, p can be a positive integer.
[0446] In addition, for example, a first / second inverse transform kernel can be determined based on the distribution of multiple directional intensities.
[0447] Specifically, for example, the directional mode M with the greatest directional strength 1st The intensity of S(M 1st ) is smaller than the threshold TH1, and the above S(M 1st) and the second largest directional mode M 2nd The intensity of S(M 2nd Difference from ) (S diff = S(M 1st ) - S(M 2nd If )) is smaller than the threshold TH2, it can be determined that the intensity of the directionality is evenly distributed. For example, the two intensity values of the directionality may represent the histogram values of the top two directional modes with the largest histogram sizes among the directional modes derived through mapping to the HoG directional modes. Additionally, whether the intensity of the directionality is evenly distributed can be determined based on two or more intensity values of directionality.
[0448] Also, for example, the average of N directional intensities (S avg ) and the above S(M 1st If the difference of ) is smaller than the threshold TH3, it can be determined that the intensity of the directionality is evenly distributed. For example, the N directional intensities may represent the intensity of the top N directional modes with the largest directional intensity in the above HoG.
[0449] As described above, when the intensity of the directionality is evenly distributed, for example, because all directional directions have similar influence, a specific (inverse)transformation kernel can be determined for the horizontal and vertical directions. For example, the specific (inverse)transformation kernel may be a DCT-2 kernel.
[0450] Alternatively, for example, the directional mode of the current block can be selected or considered as DC mode or Planar mode, and the inverse transform kernel of the transform block can be determined according to the Regular intra block inverse transform kernel determination method described above.
[0451] Alternatively, for example, a kernel set (vertical, horizontal) related to the case where the intensity of directionality is evenly distributed is separately derived or predefined, such that the in-frame prediction mode of the current block, the size of the transformation block, and the above M1st , M 1st and M 2nd The difference of (|M 1st - M 2nd Information related to NSPT / LFNST flags or indexes can be derived based on at least one of |), and a first / second inverse transformation kernel can be determined based on the NSPT / LFNST flags or indexes related information.
[0452] Meanwhile, for example, the directional mode M with the greatest directional strength 1st The intensity of S(M 1st ) and the second largest directional mode M 2nd The intensity of S(M 2nd Difference from ) (S diff = S(M 1st ) - S(M 2nd If )) is greater than the threshold TH4, the strength of one direction, i.e., the above S(M 1st It can be judged that ) is dominant.
[0453] Or, for example, the average of N directional intensities (S avg ) and the above S(M 1st If the difference of ) is greater than the threshold TH5, the strength of one direction, i.e., the above S(M 1st It can be determined that ) is dominant. For example, N directional intensities may represent the intensities of the top N directional modes with large directional intensities in the above HoG.
[0454] As described above, in the case where the strength of one directionality is dominant, for example, among the groups that can be divided into N based on the difference between the two directionality modes described above, the group with a small difference in directionality modes (0 < |M 1st - M 2nd By mapping to | <= m, where m can be a positive integer), the (inverse)transformation kernel can be determined based on this. For example, the above group (IPM diff,idxThe index of ) can be determined to be 0, and a transformation kernel can be determined based on the index and LUT.
[0455] Or, for example, the directional mode of the current block is M 1st By selecting or considering it as such, the inverse transformation kernel of the transformation block can be determined according to the Regular intra block (inverse) transformation kernel determination method described above.
[0456] Alternatively, for example, a kernel set (vertical, horizontal) associated with a case where one directionality is dominant is separately derived or predefined, such that the in-frame prediction mode of the current block, the size of the transformation block, and the above M 1st , M 1st and M 2nd The difference of (|M 1st - M 2nd Information related to NSPT / LFNST flags or indexes can be derived based on at least one of |), and a first / second inverse transformation kernel can be determined based on the NSPT / LFNST flags or indexes related information.
[0457] Meanwhile, in the case where information of a first / second (inverse)transformation kernel is adaptively determined according to the distribution of direction-related information according to the above-described embodiment, the scanning direction of the (inverse)transformation coefficient can be adaptively determined based on at least one of the following information: the distribution of direction-related information, the strength of the distribution of direction-related information determined based on the distribution of direction-related information, whether the first / second (inverse)transformation kernel is applied, the type of the first / second (inverse)transformation kernel, and whether the horizontal (inverse)transformation and the vertical (inverse)transformation are identical when the first / second (inverse)transformation kernel is a 1D (inverse)transformation.
[0458] Meanwhile, regarding the transformation and inverse transformation of residual signals, an optimal transformation method related to residual signals can be proposed based on the prediction mode of the current block, directionality-related information, and reference sample-related information.
[0459] FIG. 35 shows an example of an inverse conversion procedure performed in a coding device.
[0460] Referring to FIG. 35, the coding device or the inverse transformation unit of the coding device may include a unit for checking whether to perform an inverse second transformation (3500), an inverse second transformation unit (3510), a unit for determining an inverse first transformation kernel (3520), a unit for checking whether to perform an inverse first transformation (3530), a residual feature prediction unit (3540), and an inverse first transformation unit (3550). For example, as shown in the block diagram of FIG. 35, an inverse transformation can be performed on the current block to derive a restored residual signal.
[0461] The inverse second transformation execution confirmation unit (3500) can parse information related to the inverse second transformation execution flag that determines whether to perform an inverse second transformation. Based on the parsed information related to the inverse second transformation execution flag, whether to perform an inverse second transformation can be determined. For example, if the flag indicating whether to perform an inverse second transformation is true, an inverse second transformation can be performed, and the inverse second transformation can be performed by the inverse second transformation unit (3510).
[0462] The inverse second-order transformation unit (3510) may additionally include an inverse second-order transformation kernel determination unit (not shown) and an inverse second-order transformation execution unit (not shown) related to performing the inverse second-order transformation. The inverse second-order transformation kernel determination unit may determine a transformation kernel to be used in relation to performing the inverse second-order transformation of the transformation coefficients. For example, the inverse second-order transformation kernel determination unit may determine a transformation kernel to be used when performing the inverse second-order transformation on the transformation coefficients derived by the inverse quantization unit of the coding device. In addition, the inverse second-order transformation execution unit may obtain the transformation kernel determined by the inverse second-order transformation kernel determination unit and perform the inverse second-order transformation based thereon. For example, the inverse second-order transformation execution unit may perform the inverse second-order transformation on the inverse transformation coefficients obtained from the inverse quantization unit of the coding device based on the inverse transformation kernel determined by the inverse second-order transformation kernel determination unit. In addition, the inverse first transformation unit (3550) can obtain information related to the result of the inverse second transformation.
[0463] Meanwhile, if the flag indicating whether to perform the parsed inverse second transformation is false, the inverse first transformation kernel determination step may be performed without performing the inverse second transformation. The inverse first transformation kernel determination step may be performed by the inverse first transformation kernel determination unit (3520).
[0464] The inverse first-order transformation kernel determination unit (3520) can determine a transformation kernel related to performing an inverse first-order transformation.
[0465] FIG. 36 illustrates the configuration of the inverse first-order transformation kernel determination unit (3520) as an example.
[0466] Referring to FIG. 36, the inverse first-order transformation kernel determination unit (3520) may include an inverse first-order transformation kernel induction condition verification unit (3600), an inverse first-order transformation kernel induction unit (3610), and an inverse first-order transformation kernel parsing unit (3620).
[0467] The inverse first-order transformation kernel induction condition check unit (3600) can determine whether to determine the transformation kernel by inducing information related to the inverse first-order transformation kernel based on at least one of the size of the current block and the prediction mode, or to determine the transformation kernel by parsing the information.
[0468] For example, information related to the (inverse)transformation kernel is signaled from the encoding device, and a (inverse)transformation kernel signaling flag related to whether to determine the (inverse)transformation kernel may be signaled.
[0469] For example, when the above (inverse) conversion kernel signaling flag is true, information related to the inverse conversion kernel for the current block to be performed inverse first conversion is signaled, and the inverse conversion kernel can be determined based on the information in the inverse first conversion kernel parsing unit (3620).
[0470] As another example, when the above (inverse) transformation kernel signaling flag is false, a predetermined inverse transformation kernel may be used based on whether techniques such as ISP (Intra Subblock Partition) and SBT (Sub Block Partition Transform) are applied to the current block or a block / tile / slice / CTU, etc., that will perform the inverse first transformation.
[0471] Alternatively, as another example, if the above (inverse)transformation kernel signaling flag is false, a predetermined inverse transformation kernel may be used based on the size of the current block to be performed inverse first transformation. For example, if at least one of the width or height of the current block is within a specific range, a predetermined inverse transformation kernel may be used in relation to at least one of the horizontal inverse transformation or the vertical inverse transformation. In this case, the horizontal and vertical (inverse)transformation kernels may not be the same, and there may be one or more (inverse)transformation kernels of the same direction within a single (inverse)transformation block.
[0472] Figure 37 illustrates an exemplary process of performing an inverse transformation in the horizontal direction.
[0473] Referring to FIG. 37, the current (inverse)transformation block may be divided into one or more regions. For each region, an inverse transformation may be performed based on the same kernel or different kernels. The number of regions and the location where the regions are divided may be determined based on at least one of the partitioning structure of adjacent blocks, the characteristics of neighboring reference regions, feature points derived based on the characteristics of neighboring reference regions, and the prediction mode of the current block.
[0474] For example, the surrounding reference area may be an adjacent area of a block containing the current (inverse) transform block, such as the shaded area in FIG. 37, and the reference area may be composed of one or more lines. Additionally, the feature point may indicate a specific location in the reference area associated with a feature derived based on a restored sample of the reference area or a residual signal of the reference area. In this case, the feature may represent the maximum value, minimum value, absolute value, first derivative value, second derivative value, etc. of the surrounding reference area.
[0475] For example, the feature point may be derived based on the location of the maximum value, the location of the minimum value, or the location with the largest absolute value of the residual signal within the current (inverse)transformation block or the surrounding reference area of the block containing the current (inverse)transformation block. In this case, if there is one or more locations of the maximum value, the minimum value, or the location with the largest absolute value of the residual signal, a feature point list may be derived. Among the feature point candidates in the feature point list, the feature point candidate with the smallest variance of the surrounding area brightness value or sample value may be determined as the feature point. If there are two or more candidates with the smallest variance of the surrounding area brightness value or sample value among the feature point candidates, the feature point candidate located at the top and the leftmost may be determined as the feature point. Alternatively, the feature point candidate corresponding to index 0 of the feature point list may be determined as the feature point.
[0476] As another example, the feature point may be derived based on the first derivative value or the second derivative value in the surrounding reference region of the current (inverse)transformation block or the block containing the current (inverse)transformation block. For example, the feature point may be derived based on the location where the absolute value of the first derivative value or the absolute value of the second derivative value is the largest. In this case, if there is one or more locations where the absolute value of the first derivative value or the absolute value of the second derivative value is the largest, a feature point list may be derived. Among the feature point candidates in the feature point list, the feature point candidate with the smallest variance of the surrounding region brightness value or sample value may be determined as the feature point. If there are two or more feature point candidates with the smallest variance of the surrounding region brightness value or sample value, the feature point candidate located at the top and leftmost may be determined as the feature point.
[0477] Meanwhile, if the prediction mode of the block containing the current (inverse)transformation block is intra prediction, the number of regions and the locations where the regions are divided can be determined based on feature points and directionality.
[0478] Figure 38 illustrates an exemplary region segmentation when the prediction mode of the current block is in-frame prediction.
[0479] Referring to FIG. 38, the current (inverse)transformation block or the block containing the current (inverse)transformation block may be divided into one or more regions. For each region, an inverse transformation may be performed based on the same kernel or different kernels. The number of regions and the location where the regions are divided may be determined based on at least one of the prediction mode, feature points, and directionality of the current block.
[0480] For example, if the prediction mode of a block containing the current (inverse)transformation block is a directionality-based intra-frame prediction mode, the region may be divided based on the location or range associated with the directionality-based intra-frame prediction mode and the feature points. Specifically, for example, during the process of generating a prediction block, the region may be divided based on the range influenced by samples of feature point locations within a surrounding reference area. In this case, if the current block does not have a directionality, a directionality derived based on MPM (Most Probable Mode), HoG (Histogram of Gradient), etc., may be used as the directionality of the current block.
[0481] Additionally, a surrounding reference area for deriving feature points may be determined based on the orientation or derived orientation of the block containing the current (inverse)transformation block and the size, shape, or ratio of the block containing the (inverse)transformation block. For example, a surrounding reference area for deriving feature points may be determined based on the width and height ratio of the current block.
[0482] For example, if the orientation of a block containing the current (inverse)transformation block or the derived orientation is closer to the horizontal direction than the vertical direction, the region may be determined based on the range influenced by the sample of feature points derived based on the surrounding reference region adjacent to the left of the block containing the current (inverse)transformation block. For example, if the difference between the mode number of the in-frame prediction mode based on the orientation of the current block or the in-frame prediction mode related to the derived orientation and the mode number of the horizontal direction prediction mode is k or less, the orientation of the current block or the derived orientation may be determined to be closer to the horizontal direction than the vertical direction.
[0483] As another example, if the orientation of a block containing the current (inverse)transformation block or the derived orientation is closer to the vertical direction than the horizontal direction, the region can be determined based on the range influenced by samples of derived feature points based on the surrounding reference region adjacent to the upper side of the block containing the current (inverse)transformation block. For example, if the difference between the mode number of the in-frame prediction mode based on the orientation of the current block or the in-frame prediction mode related to the derived orientation and the mode number of the vertical direction prediction mode is k or less, the orientation of the current block or the derived orientation can be determined to be closer to the vertical direction than the horizontal direction.
[0484] Alternatively, for example, if the prediction mode of a block containing the current (inverse)transformation block is inter-prediction, the number of regions and division positions of the current block may be determined based on the number of regions and division positions of the reference block. Specifically, for example, if the residual signal of the reference block derived based on motion vectors is transformed based on an optimal transformation method, the number of regions and division positions of the reference block may be determined as the number of regions and division positions of the block containing the current (inverse)transformation block.
[0485] Alternatively, for example, if intra-sub-partitions such as ISPs are applied to a block containing the current (inverse)transformation block, the number of regions of the block containing the current (inverse)transformation block may be determined based on the number of divisions of the block to which intra-sub-partitions are applied. For example, the number of regions of the block containing the current (inverse)transformation block may have the same number of regions as the number of divisions of the block to which intra-sub-partitions are applied. Furthermore, the location where the region is divided within the current block may be determined based on the location divided by the intra-sub-partitions. For example, the location where the region is divided within the current block may be determined to be the same location as the location divided by the intra-sub-partitions. Alternatively, as another example, the number and location of the regions to be divided may be determined based on a feature point, which is the location where the features of the surrounding reference region are maximum or minimum. In this case, the features of the surrounding reference region may represent at least one of the features described above.
[0486] Alternatively, for example, if a sub-block partitioning transformation, such as SBT (Sub Block Partition Transform), is applied to a block containing the current (inverse)transform block, the number of regions in the block containing the current (inverse)transform block may be determined based on the number of divisions of the block to which the sub-block partitioning transformation is applied. For example, the number of regions in the block containing the current (inverse)transform block may have the same number of regions as the number of divisions of the block to which the sub-block partitioning transformation is applied. Furthermore, the location where the region is partitioned in the current block may be determined based on the location partitioned by the sub-block partitioning transformation. For example, the location where the region is partitioned in the current block may be determined to be the same location as the location partitioned by the sub-block partitioning transformation. Alternatively, as another example, the number and location of the partitioned regions may be determined based on feature points, which are the locations where the features of the surrounding reference region are maximum or minimum.
[0487] At this time, the characteristics of the surrounding reference area may represent at least one of the characteristics described above, such as the maximum value, minimum value, absolute value, first derivative value, second derivative value of the surrounding reference area sample.
[0488] Meanwhile, the kernel determined by deriving or parsing conversion kernel-related information in the inverse first-order conversion kernel derivation unit (3610) or the inverse first-order conversion kernel parsing unit (3620) may be one or more kernels. At this time, the one or more kernels determined may be the same or different from each other. Alternatively, the one or more kernels may be determined based on index-related information regarding a combination of one or more kernels.
[0489] Meanwhile, the inverse first-order transformation check unit (3530) can output the input signal as a restored residual signal if the inverse first-order transformation kernel determined by the inverse first-order transformation kernel determination unit (3520) means skip. Alternatively, if the inverse first-order transformation kernel does not mean skip, a residual feature prediction step and an inverse first-order transformation step for predicting residual features may be performed. The residual feature prediction step and the inverse first-order transformation step may be performed by the residual feature prediction unit (3540) and the inverse first-order transformation unit (3550), respectively.
[0490] Meanwhile, for example, based on whether the residual feature prediction step performed by the residual feature prediction unit (3540) is performed, the (inverse)quantization method and (inverse)quantization parameters, etc., may be determined during the (inverse)quantization process. As an example, (inverse)quantization may be performed based on predefined (inverse)quantization parameters based on whether the residual feature prediction step is performed.
[0491] FIG. 39 illustrates the configuration of a residual feature prediction unit (3540) in an exemplary manner.
[0492] Referring to FIG. 39, the residual feature prediction unit (3540) may include a residual feature prediction condition verification unit (3900) and a residual feature prediction execution unit (3910).
[0493] The residual feature prediction condition verification unit (3900) can determine whether to perform the residual feature prediction execution unit (3910). For example, the residual feature prediction condition verification unit (3900) can determine whether to perform residual feature prediction based on information indicating whether residual feature prediction is available or information indicating whether residual feature prediction is performed.
[0494] Specifically, for example, the residual feature prediction condition check unit (3900) can determine whether to perform residual feature prediction based on the residual feature prediction enabled flag and the residual feature prediction execution flag.
[0495] The above residual feature prediction enabled flag may be signaled from an encoding device, and the unit to which the residual feature prediction enabled flag is transmitted may be a unit such as a CU, slice, tile, picture, CTU, sequence, frame, etc.
[0496] Additionally, the residual feature prediction execution flag may be derived based on at least one of the features of the surrounding reference region of the block containing the current (inverse)transformed block, the partition structure of the adjacent block, the prediction mode of the current block, the size or shape of the current block, and the prediction mode of the adjacent restored block. Alternatively, the residual feature prediction execution flag may be signaled from an encoding device. In this case, the unit to which the residual feature prediction execution flag is transmitted may be a unit such as a CU, slice, tile, picture, CTU, sequence, frame, etc.
[0497] At this time, the characteristics of the surrounding reference area may represent at least one of the characteristics described above, such as the maximum value, minimum value, absolute value, first derivative value, second derivative value of the surrounding reference area sample.
[0498] The residual feature prediction unit (3910) can predict residual features. For example, the residual features predicted by the residual feature prediction unit (3910) can be used as input to the inverse first transformation unit (3550).
[0499] The above residual feature may represent information related to the residual signal. For example, the above residual feature may be information that represents the shape of the residual signal.
[0500] Figure 40 illustrates the residual features in an exemplary manner.
[0501] Residual features can be represented based on slope and intercept. Alternatively, residual features can be represented based on angle and distance from the center. Or, residual features can be represented based on the location of the center point and the size of the radius.
[0502] For example, if the absolute value of the maximum or minimum value of the first derivative of the surrounding reference region of the block containing the current (inverse) transformation block is smaller than a value derived based on a predefined value or signaled information, the residual feature prediction step performed by the residual feature prediction execution unit (3910) may not be performed.
[0503] As another example, if the prediction mode of a restoration block adjacent to the current block is a non-directional mode such as DC mode or PLANAR mode among the in-screen prediction modes, the residual feature prediction step performed by the residual feature prediction execution unit (3910) may not be performed.
[0504] Meanwhile, residual features may be predefined. Alternatively, residual features may be derived based on the current block or surrounding reference regions. Alternatively, information related to residual features may be signaled.
[0505] For example, the residual features predicted by the residual feature prediction unit (3910) can receive information related to correction and perform correction. The information related to correction may have one or more values and may be applied by row or by column.
[0506] Meanwhile, the residual feature prediction execution unit (3910) can perform residual feature prediction based on the prediction mode of the block including the current (inverse) transform block and the features of the surrounding reference region. For example, the features of the surrounding reference region may represent at least one of the features described above, such as the maximum value, minimum value, absolute value, first derivative value, second derivative value of the surrounding reference region sample.
[0507] Figure 41 exemplarily illustrates residual features when the prediction mode of the current block is in-frame prediction.
[0508] Referring to FIG. 41, when the prediction mode of a block containing a current (inverse)transform block is an in-frame prediction mode, residual features can be derived based on the in-frame prediction mode and surrounding reference samples of the current block. For example, the location of a feature point in a reference region related to the generation of a prediction signal of a block containing a current (inverse)transform block is used as intercept-related information, and information related to the orientation of a block containing a current (inverse)transform block is used as gradient-related information, and residual features can be derived based on the intercept-related information and the gradient-related information. The orientation of the current block can be signaled based on the prediction mode of the current block, or can be derived based on the prediction mode, reference samples, etc.
[0509] Alternatively, for example, residual features can be derived based on the residual signal of the block corresponding to the current (inverse)transform block of the reference picture. For example, residual features can be derived based on the residual signal at the same location as the current (inverse)transform block of the reference picture.
[0510] Figure 42 exemplarily illustrates residual features when the prediction mode of the current block is an inter-frame prediction mode.
[0511] Referring to FIG. 42, when the prediction mode of a block containing the current (inverse)transformation block is an inter-frame prediction mode, residual features can be derived based on the reference block. For example, if information regarding residual features of a reference block based on motion vectors exists, residual features of a block containing the current (inverse)transformation block can be derived based on said information regarding residual features.
[0512] Alternatively, for example, the location of a feature point in the reference region of a block containing the current (inverse)transform block may be used as intercept-related information, and information related to the orientation of the block containing the current (inverse)transform block may be used as gradient-related information, so that residual features can be derived based on the intercept-related information and the gradient-related information. The orientation of the current block may be derived based on a reference sample or a reference block.
[0513] Alternatively, for example, residual features of a block containing the current (inverse)transformation block can be derived based on the prediction mode of a reference block. In this case, if there is not a single prediction mode within the reference block, residual features of the current block can be derived by considering the in-frame prediction mode having directionality derived based on HoG (Histogram of Gradient) or MPM (most probable mode).
[0514] Specifically, for example, if the prediction mode of the reference block is an in-frame prediction mode, the directionality of the block containing the current (inverse)transformation block can be derived based on the directionality of the reference block. For example, the directionality of the block containing the current (inverse)transformation block can be derived to be the same as the directionality of the reference block. If the directionality of the reference block does not exist, the directionality can be derived based on the derived directionality of the reference block, the derived directionality of the block containing the current (inverse)transformation block, or the weighted sum of the directions of the two blocks.
[0515] Figure 43 exemplarily illustrates residual features when the prediction mode of the current block is a geometric partitioning prediction mode.
[0516] Referring to FIG. 43, when the prediction mode of a block containing the current (inverse)transformation block is a geometric partition prediction mode, residual features can be derived based on the partition locations. For example, geometric partition locations can be used as residual features.
[0517] Additionally, for example, residual features can be corrected based on a combination of geometric partitioning prediction modes. Specifically, if the prediction modes forming the combination of geometric partitioning prediction modes are identical, the partitioning location can be used as a residual feature. Alternatively, if the prediction modes forming the combination of geometric partitioning prediction modes are different, the residual feature can be derived from the partitioning location to a location corrected by a distance d in the direction of the area where the prediction mode is located. In this case, the said d can be derived based on information related to the mixed area of the two prediction blocks during the geometric partitioning prediction mode process.
[0518] Figure 44 exemplarily illustrates the residual features when in-screen subdivision is applied to the current block.
[0519] Referring to FIG. 44, when an in-screen subdivision is applied to a block containing a current (inverse) transformation block, the current block may be divided into one or more in-screen subdivision blocks. At this time, based on the order of the in-screen subdivision blocks, a residual feature for each in-screen subdivision unit may be derived.
[0520] For example, in the case of the first block of an in-screen subdivision, the residual features of the first block of an in-screen subdivision can be derived based on the location of feature points in the surrounding reference area of the first block of an in-screen subdivision and the directionality of the in-screen subdivision.
[0521] In addition, for example, if it is not the first block of a sub-division within the screen, the residual features of the current sub-division block within the screen can be derived based on the feature point locations of the surrounding reference areas of the current sub-division block within the screen and the directionality of the sub-division within the screen.
[0522] Figure 45 exemplarily illustrates residual features when a sub-block transformation is applied to the current block.
[0523] Referring to FIG. 45, when a sub-block transformation is applied to a block containing a current (inverse)transformation block, the current block may contain one or more blocks. For example, the residual feature of the (inverse)transformation execution block corresponding to the region where the (inverse)transformation is performed in the block containing the current (inverse)transformation block may be derived based on at least one of the surrounding reference region, directionality, and reference block according to the motion vector of the (inverse)transformation execution block.
[0524] For example, residual features can be derived based on the aforementioned surrounding reference region and the directionality. The surrounding reference region may represent at least one of the following: a surrounding reference region of an inverse transformation block where an inverse transformation is performed in the current block; a surrounding reference region of a reference block area identical to the inverse transformation block in a reference block based on the motion vector of the current block; or a region obtained by weighted summing the two types of surrounding reference regions. Additionally, the directionality may be derived as the directionality of the current block based on MPM (Most Probable Mode) and HoG (Histogram of Gradient), and if a directionality of the reference block based on the motion vector exists, the directionality of the reference block may be derived as the directionality of the current block.
[0525] As another example, residual features can be derived based on the residuals of the same reference block region as the block performing the inverse transformation in the reference block according to the motion vector of the current block.
[0526] Meanwhile, an inverse first-order transformation step can be performed based on the predicted residual features, and the inverse first-order transformation step can be performed by an inverse first-order transformation unit (3550).
[0527] FIG. 46 illustrates the configuration of the inverse first conversion unit (3550) as an example.
[0528] Referring to FIG. 46, the inverse first-order transformation unit (3550) may include an inverse first-order transformation execution unit (4600) and an inverse space transformation execution unit (4610). Some or all of the operations of the inverse space transformation execution unit (4610) and the inverse first-order transformation execution unit (4600) may be omitted, and the execution order of the inverse space transformation execution unit (4610) and the inverse first-order transformation execution unit (4600) may be changed. Additionally, the inverse first-order transformation unit (3550) may derive a residual signal by performing an inverse first-order transformation on the transformation coefficients based on the transformation kernel. The residual signal may be used as an input to the in-loop filter unit of the coding device.
[0529] The inverse first transformation execution unit (4600) can derive a residual signal by performing an inverse first transformation based on the inverse first transformation kernel and transformation coefficients. At this time, when the residual feature prediction step performed by the residual feature prediction unit (3540) described above is performed, the scan order of the (inverse) transformation coefficients can be determined based on the residual features.
[0530] For example, if a residual feature prediction step is performed, a predefined (inverse)transform coefficient scan sequence based on residual features may be applied.
[0531] The inverse space transformation execution unit (4610) can derive a restored inverse space transformation by performing an inverse space transformation on a residual signal or an inverse transformation kernel. At this time, the degree of inverse space transformation (K) can represent distance, angle, the position of a reference point, etc., depending on the inverse space transformation method.
[0532] Information related to (in)space transformation can be signaled, or derived based on the size of the current block, parameters of surrounding blocks, etc. The parameters of the surrounding blocks may include the (intra / inter) prediction mode of the surrounding blocks, information related to space transformation, etc. For example, the information related to space transformation may include flag information indicating whether space transformation is applied, index information of the space transformation, information related to the type of space transformation, information related to the distance of the space transformation, information related to the angle of the space transformation, information related to the set of space transformation candidates, information related to the space transformation candidates, etc.
[0533] For example, a set of spatial transformation candidates may be constructed based on the (intra / inter) prediction mode of the surrounding blocks. Additionally, a set of spatial transformation candidates for the current block may be derived based on information related to the spatial transformation of the surrounding blocks (or candidates for spatial transformation). The set of spatial transformation candidates may include n candidates. In this case, index information representing one of the n candidates may be signaled.
[0534] Additionally, for example, a flag indicating whether a spatial transformation is applied can be signaled at the coding block / unit level, and index information or information related to the type / distance / angle of the spatial transformation can be signaled at the transformation block / unit level.
[0535] Additionally, for example, flag information indicating whether a spatial transformation is applied may be signaled first, and if the value of the flag is 1, information related to the spatial transformation type may be signaled. Information related to the spatial transformation distance or information related to the angle may be signaled after the information related to the spatial transformation type has been signaled. The information related to the spatial transformation distance or the angle may indicate the spatial transformation distance or angle differently based on the value of the information related to the spatial transformation type.
[0536] Meanwhile, after performing the inverse first-order transformation, the inverse space transformation may be performed. Alternatively, the inverse space transformation may be performed on the (inverse) transformation kernel before the inverse first-order transformation is performed. For example, the inverse space transformation may represent a shift, rotate, flip, point symmetry, etc. Or, for example, the inverse space transformation may represent a 1D or 2D unit transformation based on matrix operations.
[0537] Figure 47 illustrates an inverse space transformation.
[0538] Referring to FIG. 47, the inverse space transformation can represent transformations related to shift, rotate, flip, and point symmetry.
[0539] For example, in the case of shift, different inverse spatial transformation degrees (K) may be used for each row or column. The inverse spatial transformation degree (K) may represent the (shift) distance in the horizontal or vertical direction.
[0540] Specifically, for example, referring to an embodiment related to shifting in FIG. 47, for the first row, the degree of inverse space transformation (K) may be (block size-1). For the second row, it may have half the degree of inverse space transformation (K) of the first row, and for the last row, the degree of inverse space transformation (K) may be 0.
[0541] Figure 48 illustrates an exemplary method for deriving the degree of inverse space transformation (K).
[0542] Referring to FIG. 48, the inverse space transformation degree (K) for each row or column can be derived based on residual features and transformation kernel coefficients. For example, the inverse space transformation degree (K) can be derived based on residual features and the horizontal or vertical distance between the position of the maximum value where the second derivative of the transformation kernel coefficient is negative or the position of the minimum value where the second derivative is positive.
[0543] Meanwhile, a shift method may be determined depending on the target of the inverse space transformation. For example, if the target of the inverse space transformation is a residual signal, a cyclic shift may be performed as in the shift example of FIG. 47. As another example, if the target of the inverse space transformation is a (inverse)transform kernel based on a periodic signal, a periodic signal shift used in the cyclic process may be performed in relation to the (inverse)transform coefficient of the corresponding period.
[0544] Meanwhile, the embodiment related to rotation in FIG. 47 illustrates a case where a transformation is performed by rotating 90 degrees clockwise. The degree of rotation can be derived based on residual features and a transformation kernel. Additionally, the degree of inverse space transformation (K) can represent the angle of rotation clockwise.
[0545] Meanwhile, the embodiment related to the flip in FIG. 47 illustrates an example in which a left-right flip transformation is performed based on the center. The criteria related to performing the flip transformation can be determined based on residual features and a transformation kernel. Additionally, the inverse space transformation degree (K) can indicate the position of the reference point for the flip.
[0546] Meanwhile, the embodiment related to point symmetry in FIG. 47 may be an example in which a point symmetry transformation is performed with respect to the center. The criteria related to performing the point symmetry transformation may be determined based on residual features and a transformation kernel. Additionally, the inverse space transformation degree (K) may indicate the reference point location for performing the point symmetry transformation.
[0547] Meanwhile, the inverse space transformation method may be predefined, or whether to perform the inverse space transformation based on a certain inverse space transformation method may be determined based on residual features and an inverse first-order transformation kernel.
[0548] Alternatively, information related to the inverse space transformation method may be signaled. For example, the index of a list of inverse space transformation candidates derived based on residual features and transformation kernels may be signaled.
[0549] Meanwhile, the residual signal derived by the inverse space transformation unit (4610) can be input to the in-loop filter unit of the coding device together with the prediction signal.
[0550] Meanwhile, the prediction unit of the coding device can generate a prediction signal by performing intra-frame prediction based on the restored signals around the current block. The intra-frame prediction can be performed based on a prediction signal generated based on intra-frame prediction or inter-frame prediction of an area located around the block where the current prediction is performed, and a residual signal derived from the inverse transformation unit.
[0551] The prediction mode related to the above-mentioned intra-frame prediction can be determined through parsing or determined according to a series of processes during the coding process. For intra-frame prediction, a single directional prediction mode, PAETH mode, DC mode, smooth mode, recursive prediction mode, or a prediction mode based on inter-component correlation (e.g., CfL (Chroma from luma), CCLM (Cross-component linear model), MHCCP (Multi-hypothesis cross-component prediction), CCCM (Convolutional cross-component model), etc.) may be used.
[0552] As an example of in-frame prediction, a prediction signal for the current block can be generated through a prediction based on directionality. For example, the prediction signal can be generated based on a signal generated based on a restoration signal of the periphery of the block where the current prediction is performed, and based on directionality. In this case, the signal generated based on the restoration signal can be generated through interpolation, and the interpolation method can be diverse. For example, a signal that is not an integer position can be generated through linear interpolation. In addition, the directionality can be diverse, for example, as shown in FIG. 8 described above.
[0553] Additionally, for example, in the case of intra-frame prediction, during the process of generating a prediction signal through direction-based prediction, the prediction signal may be generated based on the position within the block of the prediction signal to be generated. In this case, the vertical or horizontal position of each signal may be in integer units or fractional units between integer units.
[0554] Meanwhile, the prediction unit of the coding device can generate a prediction signal by performing inter-frame prediction. For example, based on prediction-related information, a prediction signal can be generated based on one or more prediction-related signals generated through motion compensation.
[0555] Meanwhile, the filtering unit of the coding device can perform filtering procedures such as in-loop filtering. For example, in-loop filtering can be performed based on in-loop filter information and filter coefficients. In-loop filtering can not only provide visual quality improvement but also enhance compression efficiency.
[0556] For example, whether to perform filtering on a block basis, a sub-block basis, or a pixel basis can be determined.
[0557] The filtering section of the coding device may further include an in-loop filter pipeline, and the in-loop filter pipeline may include a loop filter section, a cross-component sample offset, a constrained directional enhancement filter section, a super resolution section, and a loop restoration section.
[0558] For example, the loop filter unit may perform loop filtering, such as deblocking. In relation to the loop filter, the process may include a step of determining whether to perform filtering, a step of determining the filter size, a step of determining the filter strength, and a step of performing the filter. This may be performed in blocks.
[0559] For example, information related to the sampling offset between color components can be derived from the sampling offset section between color components. The process of deriving the sampling offset between color components can be performed in block units.
[0560] For example, the limited directionality enhancement filter unit may perform limited directionality enhancement filter-related operations based on a loop-filtered image. The limited directionality enhancement filter-related operations may include a directionality determination step, a filter shape determination step, a filter intensity determination step, and a filter execution step. This may be performed in block units, specifically in the form of a limited directionality enhancement filter application area block (LFB).
[0561] Additionally, for example, whether to perform the operation related to the restricted directional enhancement filter mentioned above can be determined based on flag information such as CDEF_enable_flag or CDEF_frame_flag, which can be parsed from SPS or PPS. Specifically, for example, if the execution status of the operation related to the restricted directional enhancement filter in the current frame is true, the execution status of the operation related to the restricted directional enhancement filter for each application area block of the operation related to the restricted directional enhancement filter can be parsed at the LFB level based on flags such as cdef_flag_parsing. Parameters to be used in the LFB can be parsed, the filter shape can be derived, or it can be determined through signaling. Furthermore, the coefficients of the restricted directional enhancement filter can be derived, and filtering can be performed.
[0562] For example, the super-resolution unit can generate super-resolution image information by performing upscaling. The upscaling is performed to restore resolution when downscaling was performed during the encoding stage, and can be performed in the order of horizontal direction and vertical direction. Upscaling can be performed in block units.
[0563] For example, the loop restoration unit may perform loop restoration to generate a loop-restored image. The loop restoration may include a restoration type determination step and a restoration execution step. Additionally, the restoration type of the loop restoration unit may include a separable wiener, a self-guided, non-separable filter, a pixel classification-based wiener filter, etc. The loop restoration may be performed in block units.
[0564] The image encoding method and image decoding method according to the embodiment may include the various embodiments described above, and may include, for example, the following.
[0565] FIG. 49 schematically illustrates a video / image encoding method according to an embodiment(s) of the present disclosure. The method disclosed in FIG. 49 may be performed by an encoding device. Specifically, for example, S4900 of FIG. 49 may be performed by a prediction unit (120) of an encoding device (100). S4910 of FIG. 49 may be performed by a residual processing unit (130) of an encoding device (100). S4920 of FIG. 49 may be performed by an entropy encoding unit (140) of an encoding device (100). The method disclosed in FIG. 49 may include the embodiments described above in the present disclosure.
[0566] Referring to FIG. 49, the encoding device generates a prediction block and prediction-related information for the current block (S4900).
[0567] The above prediction-related information may include a prediction type. Additionally, the above prediction type may include one of the prediction types described above. For example, the above prediction type may include the above-described TIMD type or TIMD merge type.
[0568] Meanwhile, the encoding device can derive a template for the current block. The template may include a left template and a top template. The left template may be located around the left side of the current block, and the top template may be located around the top side of the current block. n intra prediction modes may be derived based on the template and reference samples of the template. The reference samples of the template may include first reference samples of the left reference region and second reference samples of the top reference region. The left reference region may be located around the left side of the left template, and the top reference region may be located around the top side of the top template.
[0569] Additionally, the encoding device can derive candidate intra prediction modes. The procedure for deriving the candidate intra prediction modes can be performed in the search mode determination unit of FIG. 11. For example, the candidate intra prediction modes may include a prediction mode defined according to the encoder / decoder agreement, a mode within the MPM list, or an undirected mode. The undirected mode may be a Planar mode or / and a DC mode, etc., and the predefined prediction mode may be determined based on the block size and aspect ratio, etc.
[0570] Additionally, the encoding device can calculate the cost for each of the candidate intra prediction modes. The cost for a candidate intra prediction mode can be calculated based on various cost functions including SAD, SATD, or MR SAD. For example, the cost for a candidate intra prediction mode can be calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction mode.
[0571] For example, the cost for the above candidate intra prediction mode can be calculated using the cost of the left template when the above candidate intra prediction mode has horizontal directionality. For example, if the difference between the prediction mode number of the above candidate intra prediction mode and the prediction mode number of the horizontal direction prediction mode is k or less, the above candidate intra prediction mode can be determined to have horizontal directionality.
[0572] For example, the cost for the above candidate intra prediction mode can be calculated using the cost of the upper template when the above candidate intra prediction mode has a vertical direction. For example, if the difference between the prediction mode number of the above candidate intra prediction mode and the prediction mode number of the vertical direction prediction mode is k or less, the above candidate intra prediction mode can be determined to have a vertical direction.
[0573] For example, the cost for the above candidate intra prediction mode is derived based on the cost of the above left template, the cost of the above top template, and the template weight, and the template weight may be derived based on the interval information to which the prediction mode number of the above candidate intra prediction mode belongs. Meanwhile, the above template weight includes a first weight and a second weight, and the interval information may be represented based on the first group information or the first group index.
[0574] For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the first interval, the first weight may be 2 and the second weight may be 2. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the second interval, the first weight may be 3 and the second weight may be 1. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the third interval, the first weight may be 4 and the second weight may be 0. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the fourth interval, the first weight may be 1 and the second weight may be 3. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the fifth interval, the first weight may be 0 and the second weight may be 4. If the candidate intra prediction mode is a non-directional prediction mode, the candidate intra prediction mode may be determined to belong to the first interval.
[0575] For example, the first weight may be applied to the cost of the top template, and the second weight may be applied to the cost of the left template.
[0576] Additionally, the encoding device can derive n intra prediction modes based on the above costs. The n intra prediction modes are derived into m groups, and the m groups may include at least one of an upper group, a left group, and a non-directed group. The groups of the intra prediction modes can be derived based on the costs of the left template and the upper template of the current block for the intra prediction modes.
[0577] For example, if a prediction mode has a horizontal directionality or is close to a horizontal prediction mode, the group corresponding to that prediction mode may be derived as a left group or a non-directional group. In this case, the top group may not be considered.
[0578] For example, if the prediction mode has a horizontal direction or is close to a horizontal prediction mode and the cost of the top template is calculated low, and the mode direction and group classification differ, the group for that prediction mode can be derived as, for example, a top group or a non-directional group based on the cost of the top template or the left template. Alternatively, as another example, the group can be changed to a non-directional group.
[0579] For example, if the prediction mode has a horizontal directionality or is close to a horizontal prediction mode and the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a first threshold, the group of the prediction mode may be derived as either an upper group or a non-directional group based on whether the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a second threshold. Meanwhile, the first threshold and the second threshold may be any positive real numbers.
[0580] For example, if a prediction mode has a vertical orientation or is close to a vertical prediction mode, the group corresponding to that prediction mode may be derived as an upper group or a non-directional group. In this case, the left group may not be considered.
[0581] For example, if the prediction mode has a vertical orientation or is close to a vertical prediction mode and the cost of the left template is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode can be derived as a left group or a non-directional group based, for example, on the cost of the top template or the left template. Alternatively, as another example, the group can be changed to a non-directional group.
[0582] For example, if the prediction mode has a vertical directionality or is close to a vertical prediction mode and the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a first threshold, the group of the prediction mode may be derived as either a left group or a non-directional group based on whether the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a second threshold. Meanwhile, the first threshold and the second threshold may be any positive real numbers.
[0583] Meanwhile, the above-mentioned left template may include an upper-left template and a lower-left template, and the above-mentioned top template may include an upper-left template and an upper-right template. In this case, the cost of the above-mentioned left template may be calculated based on the cost of the upper-left template and the cost of the lower-left template, and the cost of the above-mentioned top template may be calculated based on the cost of the upper-left template and the cost of the upper-right template.
[0584] For example, the cost for the above candidate intra prediction mode can be calculated using at least one of the above top-left template, above bottom-left template, above top-left template, and above-right template based on the directionality or prediction mode number of the above candidate intra prediction mode.
[0585] For example, if the prediction mode has a horizontal direction or is close to a horizontal prediction mode, and the sum of the costs of the top-left and top-right templates is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode may be derived as, for example, an upper group or a non-directional group based on a comparison of the costs of the top-left template and the bottom-left template, where the sum of the costs is calculated to be high. Alternatively, as another example, the group may be changed to a non-directional group.
[0586] For example, if the prediction mode has a horizontal directionality or is close to a horizontal prediction mode, and the ratio or difference between the cost of the top template and the cost of the left template is below a threshold, the group of the prediction mode may be derived as either an upper group or a non-directional group based on the ratio or difference between the cost of the top-left template and the cost of the bottom-left template. Meanwhile, the threshold may be any positive real number. Here, the cost of the left template or the cost of the top template may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively.
[0587] For example, if the prediction mode has a vertical orientation or is close to a vertical prediction mode, and the sum of the costs of the top-left and bottom-left templates is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode may be derived as a left group or a non-directional group, for example, based on a comparison of the costs of the top-left template and the top-right template, where the sum of the costs is calculated to be high. Alternatively, as another example, the group may be changed to a non-directional group.
[0588] For example, if the prediction mode has a vertical directionality or is close to a vertical prediction mode, and the ratio or difference between the cost of the top template and the cost of the left template is below a threshold, the group of the prediction mode may be derived as either a left group or a non-directional group based on the ratio or difference between the cost of the top-left template and the cost of the top-right template. Meanwhile, the threshold may be any positive real number. Here, the cost of the left template or the cost of the top template may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively.
[0589] When a TIMD merge type is applied to the current block, some of the operation procedures performed by the encoding device described above may be omitted or replaced with other procedures. For example, n intra prediction modes and / or weights of the current block may be derived based on the TIMD merge list.
[0590] The encoding device can generate a prediction block based on the n intra prediction modes. The encoding device can generate n (intermediate) prediction blocks based on the n intra prediction modes and generate a (final) prediction block for the current block based on the n (intermediate) prediction blocks. The encoding device can generate a (final) prediction block for the current block based on a weighted sum or weight-based fusion of the n (intermediate) prediction blocks.
[0591] For example, the n intra prediction modes are derived into m groups, and if n is 2 or greater, the prediction block may be generated by performing at least one of group-based weighted sum or pixel-location-based weighted sum based on the n intra prediction modes. For example, based on the case where the number of groups m derived for the n intra prediction modes is 2 or greater, a pixel-location-based weighted sum may be performed. For example, based on the case where m is 1, the prediction block may be generated by performing the group-based weighted sum. For example, if n is 2 and m is 2, the prediction block may be generated by performing the pixel-location-based weighted sum. For example, if n is 3 and m is 2, the prediction block may be generated by performing both the group-based weighted sum and the pixel-location-based weighted sum.
[0592] Meanwhile, prediction-related information may include the various information described above. The prediction-related information may include TIMD availability flags and / or TIMD flags, etc. The prediction type of the current block may include a TIMD type. The TIMD flag may indicate whether the TIMD type is applied as the prediction type for the current block. The prediction-related information may also include a TIMD merge flag. The TIMD merge flag may indicate whether the prediction type of the current block is a TIMD merge type.
[0593] The encoding device generates residual information for the current block (S4910).
[0594] Meanwhile, residual information is information regarding residual samples. The residual information may include information regarding quantized transformation coefficients for the residual samples.
[0595] Specifically, regarding the derivation of transformation coefficients for the current block, first / second transformation kernels can be determined based on directionality-related information as described above.
[0596] In addition, regarding the conversion of the residual signal for the current block, as described above, the conversion can be performed based on information related to residual features or information related to spatial conversion.
[0597] The encoding device encodes image information including the prediction-related information and residual information (S4920). For example, the prediction-related information may include CU syntax for the current block. The image information may be referred to as video information. Additionally, the image information may include various information according to embodiments of the present disclosure.
[0598] Encoded video information can be output in the form of a bitstream. The bitstream can be transmitted to a decoding device via a network or a storage medium. For example, video data containing the bitstream can be transmitted to a decoding device by a transmission device (or transmission unit). In this case, the video data containing the bitstream can be transmitted to the decoding device via a streaming server.
[0599] In addition, as described above, the encoding device can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is intended to derive the same prediction results from the encoding device as those performed by the decoding device, thereby increasing coding efficiency. Accordingly, the encoding device can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in memory and utilize it as a reference picture for inter-prediction. As described above, an in-loop filtering procedure, etc., may be further applied to the reconstructed picture.
[0600] According to the above-described embodiment(s), when performing intra prediction based on a template for the current block, more efficient computation and prediction can be achieved by calculating the cost of the prediction mode using the directionality of the prediction mode, and the prediction accuracy of the weighted sum result can be increased by deriving groups or correcting group classification using the directionality of the prediction mode, and the prediction quality can be improved by mitigating the mismatch between the directionality of the prediction mode and the line-unit weight directionality during the weighted sum process. Additionally, unnecessary computation can be prevented by determining whether to perform a weighted sum based on group information. Therefore, according to the above-described embodiment(s), the present invention can improve intra prediction performance and efficiency by enhancing prediction accuracy and processing speed and reducing coding complexity.
[0601] In addition, according to the above-described embodiment(s), the present invention can improve the efficiency of conversion and inverse conversion without increasing the number of bits by determining a (inverse)conversion kernel optimized for the characteristics of the current conversion block based on directionality-related information derived from the restored sample and predicted signal, without additional signaling and parsing in the coding device.
[0602] In addition, according to the above-described embodiment(s), the present invention can perform optimal conversion by predicting residual features, i.e., characteristics of residual signals, based on the prediction mode, directionality, and surrounding reference area information of a block including the current conversion block in a coding device. Through this, conversion efficiency can be improved and higher image quality can be obtained for the same bit amount.
[0603] FIG. 50 schematically illustrates a video / image decoding method according to an embodiment(s) of the present disclosure. The method disclosed in FIG. 50 may be performed by a decoding device. Specifically, for example, S5000 of FIG. 50 may be performed by an entropy decoding unit (210) of a decoding device (200), and S5010 may be performed by a prediction unit (230) of a decoding device (200). Additionally, S5020 may be performed by a residual processing unit (220) of a decoding device (200), and S5030 may be performed by an addition unit (240) of a decoding device (200). The method disclosed in FIG. 50 may include the embodiments described above in the present disclosure.
[0604] Referring to FIG. 50, a decoding device obtains prediction-related information and residual information through a bitstream (S5000). The decoding device may obtain image information including the prediction-related information and residual information through the bitstream. For example, the prediction-related information may include CU syntax for the current block. The image information may be referred to as video information. Additionally, the image information may include various information according to an embodiment of the present disclosure.
[0605] The above prediction-related information may include various information described above. The above prediction-related information may include TIMD availability flags and / or TIMD flags, etc. The prediction type of the current block may include a TIMD type. The TIMD flag may indicate whether the TIMD type is applied as the prediction type for the current block. The above prediction-related information may include a TIMD merge flag. The TIMD merge flag may indicate whether the prediction type of the current block is a TIMD merge type.
[0606] The decoding device generates a prediction block for the current block (S5010). For example, the prediction block can be generated in the manner of the example below.
[0607] First, the decoding device can derive a prediction type for the current block. The prediction type includes one of the prediction types described above. The prediction type may include, for example, the TIMD type or TIMD merge type described above.
[0608] Additionally, the decoding device can derive a template for the current block. The template may include a left template and a top template. The left template may be located around the left side of the current block, and the top template may be located around the top side of the current block. n intra prediction modes may be derived based on the template and reference samples of the template. The reference samples of the template may include first reference samples of the left reference region and second reference samples of the top reference region. The left reference region may be located around the left side of the left template, and the top reference region may be located around the top side of the top template.
[0609] In addition, the decoding device can derive candidate intra-prediction modes. The procedure for deriving the candidate intra-prediction modes can be performed in the search mode determination unit of FIG. 11.
[0610] For example, the above candidate intra-prediction modes may include a prediction mode defined according to the encoder / decoder agreement, a mode within the MPM list, or an undirected mode. The undirected mode may use a Planar mode or / and a DC mode, etc., and the above defined prediction mode may be determined based on the block size and aspect ratio, etc.
[0611] The decoding device can calculate the cost for each of the candidate intra prediction modes. The cost for a candidate intra prediction mode can be calculated based on various cost functions including SAD, SATD, or MR SAD. For example, the cost for a candidate intra prediction mode can be calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction mode.
[0612] For example, the cost for the above candidate intra prediction mode can be calculated using the cost of the left template when the above candidate intra prediction mode has horizontal directionality. For example, if the difference between the prediction mode number of the above candidate intra prediction mode and the prediction mode number of the horizontal direction prediction mode is k or less, the above candidate intra prediction mode can be determined to have horizontal directionality.
[0613] For example, the cost for the above candidate intra prediction mode can be calculated using the cost of the upper template when the above candidate intra prediction mode has a vertical direction. For example, if the difference between the prediction mode number of the above candidate intra prediction mode and the prediction mode number of the vertical direction prediction mode is k or less, the above candidate intra prediction mode can be determined to have a vertical direction.
[0614] For example, the cost for the above candidate intra prediction mode is derived based on the cost of the above left template, the cost of the above top template, and the template weight, and the template weight may be derived based on the interval information to which the prediction mode number of the above candidate intra prediction mode belongs. Meanwhile, the above template weight includes a first weight and a second weight, and the interval information may be represented based on the first group information or the first group index.
[0615] For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the first interval, the first weight may be 2 and the second weight may be 2. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the second interval, the first weight may be 3 and the second weight may be 1. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the third interval, the first weight may be 4 and the second weight may be 0. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the fourth interval, the first weight may be 1 and the second weight may be 3. For example, if the interval to which the prediction mode number of the candidate intra prediction mode belongs is the fifth interval, the first weight may be 0 and the second weight may be 4. If the candidate intra prediction mode is a non-directional prediction mode, the candidate intra prediction mode may be determined to belong to the first interval.
[0616] For example, the first weight may be applied to the cost of the top template, and the second weight may be applied to the cost of the left template.
[0617] The decoding device can derive n intra prediction modes based on the above costs. The n intra prediction modes are derived into m groups, and the m groups may include at least one of an upper group, a left group, and a non-directed group. The groups of the intra prediction modes can be derived based on the costs of the left template and the upper template of the current block for the intra prediction modes.
[0618] For example, if a prediction mode has a horizontal directionality or is close to a horizontal prediction mode, the group corresponding to that prediction mode may be derived as a left group or a non-directional group. In this case, the top group may not be considered.
[0619] For example, if the prediction mode has a horizontal direction or is close to a horizontal prediction mode and the cost of the top template is calculated low, and the mode direction and group classification differ, the group for that prediction mode can be derived as, for example, a top group or a non-directional group based on the cost of the top template or the left template. Alternatively, as another example, the group can be changed to a non-directional group.
[0620] For example, if the prediction mode has a horizontal directionality or is close to a horizontal prediction mode and the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a first threshold, the group of the prediction mode may be derived as either an upper group or a non-directional group based on whether the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a second threshold. Meanwhile, the first threshold and the second threshold may be any positive real numbers.
[0621] For example, if a prediction mode has a vertical orientation or is close to a vertical prediction mode, the group corresponding to that prediction mode may be derived as an upper group or a non-directional group. In this case, the left group may not be considered.
[0622] For example, if the prediction mode has a vertical orientation or is close to a vertical prediction mode and the cost of the left template is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode can be derived as a left group or a non-directional group based, for example, on the cost of the top template or the left template. Alternatively, as another example, the group can be changed to a non-directional group.
[0623] For example, if the prediction mode has a vertical directionality or is close to a vertical prediction mode and the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a first threshold, the group of the prediction mode may be derived as either a left group or a non-directional group based on whether the ratio or difference between the cost of the top template and the cost of the left template is less than or equal to a second threshold. Meanwhile, the first threshold and the second threshold may be any positive real numbers.
[0624] Meanwhile, the above-mentioned left template may include an upper-left template and a lower-left template, and the above-mentioned top template may include an upper-left template and an upper-right template. In this case, the cost of the above-mentioned left template may be calculated based on the cost of the upper-left template and the cost of the lower-left template, and the cost of the above-mentioned top template may be calculated based on the cost of the upper-left template and the cost of the upper-right template.
[0625] For example, the cost for the above candidate intra prediction mode can be calculated using at least one of the above top-left template, above bottom-left template, above top-left template, and above-right template based on the directionality or prediction mode number of the above candidate intra prediction mode.
[0626] For example, if the prediction mode has a horizontal direction or is close to a horizontal prediction mode, and the sum of the costs of the top-left and top-right templates is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode may be derived as, for example, an upper group or a non-directional group based on a comparison of the costs of the top-left template and the bottom-left template, where the sum of the costs is calculated to be high. Alternatively, as another example, the group may be changed to a non-directional group.
[0627] For example, if the prediction mode has a horizontal directionality or is close to a horizontal prediction mode, and the ratio or difference between the cost of the top template and the cost of the left template is below a threshold, the group of the prediction mode may be derived as either an upper group or a non-directional group based on the ratio or difference between the cost of the top-left template and the cost of the bottom-left template. Meanwhile, the threshold may be any positive real number. Here, the cost of the left template or the cost of the top template may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively.
[0628] For example, if the prediction mode has a vertical orientation or is close to a vertical prediction mode, and the sum of the costs of the top-left and bottom-left templates is calculated to be low, resulting in a difference between the mode direction and the group classification, the group for that prediction mode may be derived as a left group or a non-directional group, for example, based on a comparison of the costs of the top-left template and the top-right template, where the sum of the costs is calculated to be high. Alternatively, as another example, the group may be changed to a non-directional group.
[0629] For example, if the prediction mode has a vertical directionality or is close to a vertical prediction mode, and the ratio or difference between the cost of the top template and the cost of the left template is below a threshold, the group of the prediction mode may be derived as either a left group or a non-directional group based on the ratio or difference between the cost of the top-left template and the cost of the top-right template. Meanwhile, the threshold may be any positive real number. Here, the cost of the left template or the cost of the top template may be calculated as the sum of the cost of the top-left template and the cost of the bottom-left template, and the sum of the cost of the top-left template and the cost of the top-right template, respectively.
[0630] When a TIMD merge type is applied to the current block, some steps of the operation procedure described above may be omitted or replaced with other procedures. For example, n intra prediction modes and / or weights of the current block may be derived based on the TIMD merge list.
[0631] The decoding device can generate a prediction block based on the n intra prediction modes. The decoding device can generate n (intermediate) prediction blocks based on the n intra prediction modes and generate a (final) prediction block for the current block based on the n (intermediate) prediction blocks. The decoding device can generate a (final) prediction block for the current block based on a weighted sum or weight-based fusion of the n (intermediate) prediction blocks.
[0632] For example, the n intra prediction modes are derived into m groups, and if n is 2 or greater, the prediction block may be generated by performing at least one of group-based weighted sum or pixel-location-based weighted sum based on the n intra prediction modes. For example, based on the case where the number of groups m derived for the n intra prediction modes is 2 or greater, a pixel-location-based weighted sum may be performed. For example, based on the case where m is 1, the prediction block may be generated by performing the group-based weighted sum. For example, if n is 2 and m is 2, the prediction block may be generated by performing the pixel-location-based weighted sum. For example, if n is 3 and m is 2, the prediction block may be generated by performing both the group-based weighted sum and the pixel-location-based weighted sum.
[0633] The decoding device derives a residual sample for the current block (S5020).
[0634] The decoding device can derive residual samples based on residual information obtained through the bitstream. For example, residual samples for the current block can be derived by performing (inverse)quantization and (inverse)transformation with respect to the residual information. In this case, (inverse)quantization may be omitted and (inverse)transformation may be omitted.
[0635] Specifically, regarding the derivation of residual samples for the current block, as described above, a first / second (inverse)transform kernel can be determined based on directionality-related information.
[0636] In addition, regarding the conversion of the residual signal for the current block, as described above, the conversion can be performed based on information related to residual features or information related to spatial conversion.
[0637] The decoding device derives a recovery block for the current block (S5030).
[0638] The decoding device can generate reconstructed samples based on predicted samples and residual samples of the current block. For example, the decoding device can generate the reconstructed samples for the current block based on the residual samples for the current block and the predicted samples. The residual samples for the current block can be generated based on received residual information. Additionally, the decoding device can generate a reconstructed picture including the reconstructed samples, for example. As previously described, an in-loop filtering procedure, etc., may be further applied to the reconstructed picture.
[0639] According to the above-described embodiment(s), when performing intra prediction based on a template for the current block, more efficient computation and prediction can be achieved by calculating the cost of the prediction mode using the directionality of the prediction mode, and the prediction accuracy of the weighted sum result can be increased by deriving groups or correcting group classification using the directionality of the prediction mode, and the prediction quality can be improved by mitigating the mismatch between the directionality of the prediction mode and the line-unit weight directionality during the weighted sum process. Additionally, unnecessary computation can be prevented by determining whether to perform a weighted sum based on group information. Therefore, according to the above-described embodiment(s), the present invention can improve intra prediction performance and efficiency by enhancing prediction accuracy and processing speed and reducing coding complexity.
[0640] In addition, according to the above-described embodiment(s), the present invention can improve the efficiency of conversion and inverse conversion without increasing the number of bits by determining a (inverse)conversion kernel optimized for the characteristics of the current conversion block based on directionality-related information derived from the restored sample and predicted signal, without additional signaling and parsing in the coding device.
[0641] In addition, according to the above-described embodiment(s), the present invention can perform optimal conversion by predicting residual features, i.e., characteristics of residual signals, based on the prediction mode, directionality, and surrounding reference area information of a block including the current conversion block in a coding device. Through this, conversion efficiency can be improved and higher image quality can be obtained for the same bit amount.
[0642] In the embodiments described above, methods are described based on flowcharts as a series of steps or blocks, but the embodiments are not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps as described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, and other steps may be included, or one or more steps of the flowcharts may be omitted without affecting the scope of the embodiments of the present disclosure.
[0643] The method according to the embodiments of the present disclosure described above may be implemented in the form of software, and the encoding device and / or decoding device according to the present disclosure may be included in a device that performs image processing, such as a TV, computer, smartphone, set-top box, display device, etc.
[0644] Embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable (program) instructions, such as program modules executed by a computer. The module may be stored in memory and executed by a processor. The memory may be located inside or outside the processor and may be connected to the processor by various well-known means. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and inmovable media. Additionally, a computer-readable medium may include both computer storage media and communication media. A computer storage medium includes both volatile and non-volatile, removable and inmovable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. A communication medium typically includes computer-readable instructions, data structures, program modules, or other data of modulated data signals such as carrier waves, or other transmission mechanisms, and includes any information transmission medium.
[0645] Additionally, embodiments of the present disclosure may be implemented as a computer program (or computer program product) comprising instructions executable by a computer. The computer program comprises programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language, etc. Additionally, the computer program may be recorded on a tangible computer-readable recording medium (e.g., memory, a hard disk, a magnetic / optical medium, or a Solid-State Drive (SSD), etc.).
[0646] Accordingly, embodiments of the present disclosure may be implemented by executing a computer program as described above by a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components may be connected to one another using various buses and may be mounted on a common motherboard or mounted in other suitable ways.
[0647] Here, the processor can process instructions within the computing device, such as instructions stored in memory or storage devices to display graphic information for providing a Graphic User Interface (GUI) on external input and output devices, such as a display connected to a high-speed interface. In another embodiment, multiple processors and / or multiple buses may be used appropriately with multiple memories and memory types. Additionally, the processor may be implemented as a chipset comprising chips including multiple independent analog and / or digital processors.
Claims
1. In a video decoding method performed by a decoding device, A step of obtaining prediction-related information through a bitstream; A step of deriving a prediction type for the current block based on the above prediction-related information; A step of deriving a template for the current block based on the above prediction type; A step of deriving candidate intra-prediction modes based on a template for the current block above; A step of calculating the cost for each of the above candidate intra-prediction modes; A step of deriving n intra prediction modes for the current block based on the cost of each of the above candidate intra prediction modes; and The method includes the step of generating a prediction block for the current block based on the n intra prediction modes for the current block, and The template for the current block above includes a left template and a top template, and An image decoding method characterized in that the cost for a candidate intra prediction mode is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction mode.
2. In Paragraph 1, An image decoding method characterized in that the cost for the above candidate intra prediction mode is calculated using the cost of the left template when the above candidate intra prediction mode has a horizontal directionality.
3. In Paragraph 2, An image decoding method characterized by determining that the candidate intra prediction mode has a horizontal directionality when the difference between the prediction mode number of the candidate intra prediction mode and the prediction mode number of the horizontal direction prediction mode is k or less.
4. In Paragraph 1, An image decoding method characterized in that the cost for the above candidate intra prediction mode is calculated using the cost of the above top template when the above candidate intra prediction mode has a vertical direction.
5. In Paragraph 4, An image decoding method characterized by determining that the candidate intra prediction mode has a vertical directionality when the difference between the prediction mode number of the candidate intra prediction mode and the prediction mode number of the vertical direction prediction mode is k or less.
6. In Paragraph 1, The cost for the above candidate intra prediction mode is calculated based on the cost of the above left template, the cost of the above top template, and the template weights, and The above template weights include a first weight and a second weight, and The first weight is applied to the cost of the upper template, and the second weight is applied to the cost of the left template, and An image decoding method characterized in that the above template weight is derived based on interval information to which the prediction mode number of the above candidate intra prediction mode belongs.
7. In Paragraph 6, If the interval to which the prediction mode number of the above candidate intra prediction mode belongs is the first interval, the first weight is 2 and the second weight is 2, and If the interval to which the prediction mode number of the above candidate intra prediction mode belongs is the second interval, the first weight is 3 and the second weight is 1, and If the interval to which the prediction mode number of the above candidate intra prediction mode belongs is the third interval, the above first weight is 4 and the above second weight is 0, and If the interval to which the prediction mode number of the above candidate intra prediction mode belongs is the 4th interval, the above first weight is 1 and the above second weight is 3, and An image decoding method characterized in that, when the section to which the prediction mode number of the above candidate intra prediction mode belongs is the 5th section, the first weight is 0 and the second weight is 4.
8. In Paragraph 6, Calculating the cost of the above candidate intra-prediction mode includes calculating the cost of the above candidate intra-prediction mode using the following mathematical formula, and Cost = (w A * cost A + w L * cost L ) >> 2, Here w A is the above-mentioned first weight, w L is the above second weight, cost A is the cost of the above top template, cost L A video decoding method characterized by representing the cost of the above-mentioned left template.
9. In Paragraph 1, The above n intra prediction modes are derived into m groups, and The above m groups include at least one of an upper group, a left group, and a non-directional group, and A video decoding method characterized in that the group for the intra prediction mode is derived based on the cost of the upper template and the cost of the left template.
10. In Paragraph 9, A video decoding method characterized in that, when the above intra prediction mode has horizontal directionality, the group of the above intra prediction mode is derived as either the left group or the non-directional group.
11. In Paragraph 9, A video decoding method characterized in that, when the intra prediction mode has horizontal directionality and the ratio or difference between the cost of the upper template and the cost of the left template for the intra prediction mode is less than or equal to a first threshold, the group of the intra prediction mode is derived as either the upper group or the non-directional group based on whether the ratio or difference between the cost of the upper template and the cost of the left template is less than or equal to a second threshold.
12. In Paragraph 9, A video decoding method characterized in that, when the above intra prediction mode has a vertical directionality, the group of the above intra prediction mode is derived as either the upper group or the non-directional group.
13. In Paragraph 9, A video decoding method characterized in that, when the intra prediction mode has vertical directionality and the ratio or difference between the cost of the upper template and the cost of the left template for the intra prediction mode is less than or equal to a first threshold, the group of the intra prediction mode is derived as either the left group or the non-directional group based on whether the ratio or difference between the cost of the upper template and the cost of the left template is less than or equal to a second threshold.
14. In Paragraph 9, The above left template includes a top-left template and a bottom-left template, and the above top template includes a top-left template and an top-right template, and The cost of the above left template is calculated based on the cost of the above top left template and the cost of the above bottom left template, and A video decoding method characterized in that the cost of the upper template is calculated based on the cost of the upper left template and the cost of the upper right template.
15. In Paragraph 14, A video decoding method characterized in that, when the intra prediction mode has horizontal directionality and the ratio or difference between the cost of the upper template and the cost of the left template for the intra prediction mode is less than or equal to a threshold value, the group of the intra prediction mode is derived as either the upper group or the non-directional group based on the ratio or difference between the cost of the upper-left template and the cost of the lower-left template.
16. In Paragraph 14, A video decoding method characterized in that, when the intra prediction mode has vertical directionality and the ratio or difference between the cost of the upper template and the cost of the left template for the intra prediction mode is less than or equal to a threshold value, the group of the intra prediction mode is derived as either the left group or the non-directional group based on the ratio or difference between the cost of the upper left template and the cost of the upper right template.
17. In Paragraph 9, When n is 2 or greater, the prediction block is generated by performing at least one of group-based weighted sum or pixel-location-based weighted sum based on the n intra-prediction modes, and An image decoding method characterized by performing the pixel position-based weighted sum based on the case where m is 2 or greater.
18. In Paragraph 17, An image decoding method characterized by performing the group-unit weighted sum based on the case where m is 1.
19. A video encoding method performed by an encoding device, Step of determining the prediction type for the current block; A step of deriving a template for the current block based on the above prediction type; A step of deriving candidate intra-prediction modes based on a template for the current block above; A step of calculating the cost for each of the above candidate intra-prediction modes; A step of deriving n intra prediction modes for the current block based on the cost of each of the above candidate intra prediction modes; A step of generating a prediction block for the current block based on the n intra prediction modes for the current block; A step of generating prediction-related information based on the above-determined prediction type; and It includes the step of encoding image information containing the above prediction-related information, and The template for the current block above includes a left template and a top template, and A video encoding method characterized in that the cost for a candidate intra prediction mode is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra prediction mode.
20. In a method for transmitting image data, Acquiring a bitstream generated by a video encoding method, wherein the video encoding method comprises the steps of: determining a prediction type for a current block; deriving a template for the current block based on the prediction type; deriving candidate intra prediction modes based on the template for the current block; calculating a cost for each of the candidate intra prediction modes; deriving n intra prediction modes for the current block based on the cost of each of the candidate intra prediction modes; generating a prediction block for the current block based on the n intra prediction modes for the current block; generating prediction-related information based on the determined prediction type; and encoding video information including the prediction-related information; and The method includes the step of transmitting image data including the bitstream above, The template for the current block above includes a left template and a top template, and A transmission method characterized in that the cost for a candidate intra-prediction mode is calculated using at least one of the left template and the top template based on the directionality or prediction mode number of the candidate intra-prediction mode.
