Video decoding method and device using residual model according to inter prediction, and video encoding method and device using residual model according to inter prediction

Abstract

A method and a device for decoding a video, and a method and a device for encoding a video are provided. The method for decoding a video comprises the steps of: determining that a second block is used for inter prediction of a first block, on the basis of inter prediction information of the first block obtained from a bitstream; identifying whether a residual model of the second block can be used for inter prediction of the first block; when the residual model of the second block can be used, generating the residual model of the second block on the basis of residual information between a prediction block of the second block and a reconstructed block of the second block; and determining a prediction block of the first block by using motion information of the first block, and determining a reconstructed block of the first block by applying the residual model of the second block to the prediction block of the first block.

Description

Video decoding method and apparatus using a residual model based on inter-prediction, video encoding method and apparatus using a residual model based on inter-prediction

[0001] The present disclosure relates to the field of image encoding and decoding. More specifically, it relates to an encoding and decoding method and apparatus for predicting samples of an image.

[0002] In image encoding and decoding, the image is divided into blocks, and each block is predictively encoded and predictedly decoded through inter-prediction or intra-prediction.

[0003] Inter prediction is a technique that compresses images by eliminating temporal redundancy between images. In inter prediction, blocks of the current image are predicted using a reference image. The reference block most similar to the current block can be searched within a predetermined search range in the reference image. The current block is predicted based on the reference block, and a residual block is generated by subtracting the predicted block resulting from the prediction result from the current block.

[0004] Intra prediction is a technique that compresses images by eliminating spatial redundancy within the image. Depending on the prediction mode, intra prediction generates a prediction block based on the surrounding pixels of the current block. Then, it generates a residual block by subtracting the prediction block from the current block.

[0005] Residual blocks generated through inter-prediction or intra-prediction are passed to a decoder after undergoing transformation and quantization. The decoder inversely quantizes and inversely transforms the residual blocks, and reconstructs the current block by combining the predicted block of the current block with the residual blocks. The decoder can filter the reconstructed current block to remove artifacts within it.

[0006] The present disclosure provides a video encoding method and apparatus, and a video decoding method and apparatus for generating a restoration block of a current block using a residual model of a surrounding block generated based on residual information between a prediction block and a restoration block of a surrounding block.

[0007] In one embodiment of the present disclosure, a method for decoding video is provided. The video decoding method may include a step of determining that a second block is used for inter-predicting of the first block based on inter-predicting information of the first block obtained from a bitstream. The video decoding method may include a step of identifying whether a residual model of the second block can be used for inter-predicting of the first block. When the residual model of the second block can be used, the video decoding method may include a step of generating a residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block. The video decoding method may include a step of determining the prediction block of the first block using motion information of the first block, and determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block.

[0008] In one embodiment of the present disclosure, a method for encoding video is provided. The video encoding method may include a step of identifying whether a residual model of a second block can be used for inter-predicting of a first block. The video encoding method may include a step of generating a residual model of the second block based on residual information between a prediction block of the second block and a reconstruction block of the second block when the residual model of the second block can be used. The video encoding method may include a step of determining a prediction block of the first block and determining a reconstruction block of the first block by applying the residual model of the second block to the prediction block of the first block. The video encoding method may include a step of encoding information for determining that the surrounding block used for inter-predicting of the first block is the second block.

[0009] In one embodiment of the present disclosure, a video decoding device is provided. The video decoding device may include a memory for storing instructions; and at least one processor operably coupled to the memory and comprising a processing circuit. The device for decoding the video by the at least one processor executing the instructions alone or in cooperation may determine that a second block is used for inter-predicting of the first block based on inter-predicting information of the first block obtained from the bitstream. The device for decoding the video by the at least one processor executing the instructions alone or in cooperation may identify whether a residual model of the second block can be used for inter-predicting of the first block. When the residual model of the second block can be used, the device for decoding the video by the at least one processor executing the instructions alone or in cooperation may generate a residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block. The device for decoding the video by having at least one processor execute the instructions alone or in cooperation can determine a prediction block of the first block using motion information of the first block, and determine a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

[0010] In one embodiment of the present disclosure, a video encoding device is provided. The video encoding device may include a memory for storing instructions; and at least one processor operably coupled to the memory and comprising a processing circuit. The device encoding the video by the at least one processor executing the instructions alone or in cooperation may identify whether a residual model of a second block can be used for inter-prediction of a first block. When the residual model of the second block can be used, the device encoding the video by the at least one processor executing the instructions alone or in cooperation may generate a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block. The device encoding the video by the at least one processor executing the instructions alone or in cooperation may determine a prediction block of the first block and determine a restoration block of the first block by applying the residual model of the second block to the prediction block of the first block. An apparatus that encodes the video by having at least one processor execute the instructions alone or in cooperation may determine that a second block is used for inter-prediction of the first block. An apparatus that encodes the video by having at least one processor execute the instructions alone or in cooperation may encode information for determining that the peripheral block used for inter-prediction of the first block is the second block.

[0011] In one embodiment of the present disclosure, a method for transmitting a bitstream is provided. The bitstream transmission method may include a step of identifying whether a residual model of a second block can be used for inter-predicting of a first block. The bitstream transmission method may include a step of generating a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block when the residual model of the second block can be used. The bitstream transmission method may include a step of determining a prediction block of the first block and determining a restoration block of the first block by applying the residual model of the second block to the prediction block of the first block. The bitstream transmission method may include a step of outputting a bitstream containing information for determining that a surrounding block used for inter-predicting of the first block is the second block.

[0012] FIG. 1 is a block diagram of an image decoding device according to one embodiment of the present disclosure.

[0013] FIG. 2 is a block diagram of an image encoding device according to one embodiment of the present disclosure.

[0014] FIG. 3 illustrates a process of determining at least one encoding unit by dividing a current encoding unit according to one embodiment of the present disclosure.

[0015] FIG. 4 illustrates a process of determining at least one encoding unit by dividing a encoding unit that is in the shape of a non-square according to one embodiment of the present disclosure.

[0016] FIG. 5 illustrates a process of dividing a encoding unit based on at least one of block shape information and division shape mode information according to one embodiment of the present disclosure.

[0017] FIG. 6 illustrates a method for determining a predetermined encoding unit among an odd number of encoding units according to one embodiment of the present disclosure.

[0018] FIG. 7 illustrates the order in which a plurality of encoding units are processed when a current encoding unit is divided to determine a plurality of encoding units according to one embodiment of the present disclosure.

[0019] FIG. 8 illustrates a process for determining that, in accordance with one embodiment of the present disclosure, when the encoding unit cannot be processed in a predetermined order, the current encoding unit is divided into an odd number of encoding units.

[0020] FIG. 9 illustrates a process of determining at least one encoding unit by dividing a first encoding unit according to one embodiment of the present disclosure.

[0021] FIG. 10 illustrates that, according to one embodiment of the present disclosure, the shape that can be divided is limited when a second encoding unit of a non-square shape determined by dividing a first encoding unit satisfies a predetermined condition.

[0022] FIG. 11 illustrates a process of dividing square-shaped encoding units when the divided shape mode information cannot represent division into four square-shaped encoding units according to one embodiment of the present disclosure.

[0023] FIG. 12 illustrates that, according to one embodiment of the present disclosure, the processing order between a plurality of encoding units may vary depending on the division process of the encoding unit.

[0024] FIG. 13 illustrates a process in which, according to one embodiment of the present disclosure, a encoding unit is recursively divided to determine a plurality of encoding units, and the depth of the encoding unit is determined as the shape and size of the encoding unit change.

[0025] FIG. 14 illustrates a depth and part index (hereinafter PID) for distinguishing between coding units that can be determined according to the shape and size of the coding units according to one embodiment of the present disclosure.

[0026] FIG. 15 illustrates that a plurality of encoding units are determined according to a plurality of predetermined data units included in a picture according to one embodiment of the present disclosure.

[0027] FIG. 16 illustrates encoding units that can be determined for each picture when the combination of forms in which encoding units can be divided according to one embodiment of the present disclosure is different for each picture.

[0028] FIG. 17 illustrates various forms of encoding units that can be determined based on partitioned form mode information expressed in binary code according to one embodiment of the present disclosure.

[0029] FIG. 18 illustrates another form of encoding unit that can be determined based on partitioned form mode information represented by binary code according to one embodiment of the present disclosure.

[0030] FIG. 19 is a block diagram of an image encoding and decoding system according to one embodiment of the present disclosure.

[0031] Figure 20 is a conceptual diagram showing the motion vector coding method of a decoder according to the VVC (Versatile Video Coding) standard.

[0032] Figure 21 is a diagram illustrating a Cross-Component Linear Model (CCLM) for predicting chroma blocks.

[0033] Figure 22 is a diagram illustrating the CCCM (Convolutional Cross-Component Model) for predicting chroma blocks.

[0034] Figure 23 is a diagram illustrating the Local Illumination Compensation (LIC) technology.

[0035] FIG. 24 is a block diagram of a video decoding device according to one embodiment.

[0036] FIG. 25 is a diagram illustrating a method of using a residual model of spatially adjacent candidate blocks according to one embodiment.

[0037] FIG. 26 is a diagram illustrating a method for predicting residual information of a portion of a block according to one embodiment.

[0038] FIG. 27 is a diagram illustrating a method for applying a residual model of some CUs within a CTU to the remaining CUs that have no residual components for loop filtering according to one embodiment.

[0039] FIG. 28 is a diagram illustrating a method for generating a residual model between predicted regions and restored regions within a picture according to one embodiment and applying it to regions without residual information.

[0040] FIG. 29 is a flowchart showing a video decoding method according to one embodiment.

[0041] FIG. 30 is a block diagram of a video encoding device according to one embodiment.

[0042] FIG. 31 is a flowchart showing a video encoding method according to one embodiment.

[0043] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0044] In the present disclosure, the expression "a, b and / or c" may be replaced with "at least one of a, b or c." That is, the expression "a, b and / or c" may refer to "a," "b," "c," "a and b," "a and c," "b and c," "all of a, b and c," or variations thereof.

[0045] The present disclosure is capable of various modifications and may have various embodiments, and embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure may include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the various embodiments.

[0046] The terms used in this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, the meaning of which can be understood through the relevant explanatory sections. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0047] In this disclosure, singular expressions may include plural expressions unless the context clearly indicates otherwise. In describing the embodiments, detailed descriptions of related prior art are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of this disclosure. Additionally, numbers used in the description of the embodiments (e.g., first, second, etc.) correspond to identification symbols for distinguishing one component from another.

[0048] In the present disclosure, when one component is described as being "connected" or "connected" to another component, the one component may be directly connected to or directly connected to another component, but unless specifically stated otherwise, it may be connected or connected through another component in between.

[0049] In the present disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Components expressed in the present disclosure as "part (unit)," "module," etc., may consist of two or more components combined into a single component, or a single component may be divided into two or more more subdivided components. Furthermore, each component described below may additionally perform some or all of the functions of other components in addition to its own primary function, and some of the primary functions of each component may be exclusively performed by other components.

[0050] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Alternatively, in some situations, the expression “system configured to” may include that the system is “capable of” in conjunction with other devices or components. For example, the phrase “processor configured to perform A, B, and C” may include a dedicated processor for performing the said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0051] At least one processor according to an embodiment of the present disclosure may include various processing circuits and / or multiple processors. For example, the term “processor” as used in the present disclosure, including in the claims, may include various processing circuits including at least one processor, and at least one of the at least one processor is configured to perform the various functions described herein individually and / or collectively in a distributed manner. Where “processor,” “at least one processor,” and “one or more processors” are described as being configured to perform various functions as used in the specification, these terms may include, for example, without limitation, situations in which a single processor performs. There may be situations in which some of the cited functions and other processor(s) perform other functions among the cited functions, and also situations in which a single processor can perform all the cited functions. Additionally, at least one processor may include a combination of processors that perform the various enumerated / disclosed functions, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform the various functions.

[0052] In the present disclosure, 'image' may include a picture, a still image, a frame, a video composed of a plurality of consecutive still images, or a video.

[0053] In the present disclosure, 'sample' may include data to be processed as data assigned to a sampling location of an image. For example, a sample may include pixels within a frame in a spatial region. A block may refer to a unit comprising a plurality of samples.

[0054] Hereinafter, with reference to FIGS. 1 to 19, an image encoding method and apparatus based on a tree structure encoding unit and a conversion unit according to one embodiment of the present disclosure, an image decoding method and apparatus are disclosed.

[0055] FIG. 1 illustrates a block diagram of an image decoding device (100) according to one embodiment of the present disclosure.

[0056] The video decoding device (100) may include a bitstream acquisition unit (110) and a decoding unit (120). The bitstream acquisition unit (110) and the decoding unit (120) may include at least one processor. Additionally, the bitstream acquisition unit (110) and the decoding unit (120) may include a memory that stores instructions to be executed by at least one processor.

[0057] The bitstream acquisition unit (110) can receive a bitstream. The bitstream contains information in which an image is encoded by an image encoding device (200) described later. Additionally, the bitstream can be transmitted from the image encoding device (200). The image encoding device (200) and the image decoding device (100) can be connected via wired or wireless connection, and the bitstream acquisition unit (110) can receive the bitstream via wired or wireless connection. The bitstream acquisition unit (110) can receive the bitstream from a storage medium such as an optical medium or a hard disk. The decoding unit (120) can restore the image based on information obtained from the received bitstream. The decoding unit (120) can obtain syntax elements for restoring the image from the bitstream. The decoding unit (120) can restore the image based on the syntax elements.

[0058] To explain in detail the operation of the video decoding device (100), the bitstream acquisition unit (110) can receive a bitstream.

[0059] The image decoder (100) can perform an operation of obtaining an empty string corresponding to a partitioning mode of the encoding unit from a bitstream. The image decoder (100) can also perform an operation of determining a partitioning rule for the encoding unit. Additionally, the image decoder (100) can perform an operation of partitioning the encoding unit into a plurality of encoding units based on at least one of the empty string corresponding to the partitioning mode and the partitioning rule. To determine the partitioning rule, the image decoder (100) can determine an allowable first range of the size of the encoding unit according to the ratio of the width and height of the encoding unit. To determine the partitioning rule, the image decoder (100) can determine an allowable second range of the size of the encoding unit according to the partitioning mode of the encoding unit.

[0060] In the following, the division of a encoding unit according to one embodiment of the present disclosure will be described in detail.

[0061] First, a picture may be divided into one or more slices or one or more tiles. A slice or a tile may be a sequence of one or more Coding Tree Units (CTUs). Depending on the embodiment, a slice may include one or more tiles, and a slice may include one or more Coding Tree Units. A slice containing one or more tiles may be determined within the picture.

[0062] In contrast to the Max Coding Unit (CTU), there is the Max Coding Tree Block (CTB). A Max Coding Tree Block (CTB) refers to an NxN block containing NxN samples (where N is an integer). Each color component can be divided into one or more Max Coding Tree Blocks.

[0063] When a picture has three sample arrays (sample arrays for Y, Cr, and Cb components), the maximum encoding unit (CTU) is a unit that includes a maximum encoding block for luminance samples and two corresponding maximum encoding blocks for chroma samples, and syntax structures used to encode the luminance samples and chroma samples. When a picture is a monochrome picture, the maximum encoding unit is a unit that includes a maximum encoding block for monochrome samples and syntax structures used to encode the monochrome samples. When a picture is encoded into color planes separated by color components, the maximum encoding unit is a unit that includes syntax structures used to encode the samples of the picture and the image.

[0064] A single maximum coding block (CTB) can be divided into MxN coding blocks containing MxN samples (M and N are integers). In one embodiment, a coding block may be referred to as a coding unit.

[0065] When a picture has sample arrays for Y, Cr, and Cb components, a coding unit (CU) is a unit that includes a coding block for luminance samples and two coding blocks for corresponding chroma samples, and syntax structures used to encode the luminance samples and chroma samples. When a picture is a monochrome picture, a coding unit is a unit that includes a coding block for monochrome samples and syntax structures used to encode the monochrome samples. When a picture is a picture encoded in color planes separated by color components, a coding unit is a unit that includes syntax structures used to encode the corresponding picture and image samples.

[0066] As explained above, the maximum encoding block and the maximum encoding unit are distinct concepts, and the encoding block and the encoding unit are distinct concepts. That is, the (maximum) encoding unit refers to a data structure that includes the (maximum) encoding block containing the corresponding sample and the corresponding syntax structure. However, since a person skilled in the art can understand that the (maximum) encoding unit or the (maximum) encoding block refers to a block of a predetermined size containing a predetermined number of samples, the maximum encoding block and the maximum encoding unit, or the encoding block and the encoding unit, are referred to without distinction in the following specification unless there are special circumstances.

[0067] The image can be divided into Coding Tree Units (CTUs). The size of the CTU can be determined based on information obtained from the bitstream. The shape of the CTU can be a square of equal size, but it is not limited to this.

[0068] For example, information about the maximum size of a luma-encoded block can be obtained from a bitstream. For example, the maximum size of a luma-encoded block indicated by the information about the maximum size of a luma-encoded block may be one of 4x4, 8x8, 16x16, 32x32, 64x64, 128x128, or 256x256.

[0069] For example, information regarding the maximum size of a two-divisionable luma encoding block and the difference in luma block size can be obtained from a bitstream. The information regarding the difference in luma block size may indicate the size difference between the maximum luma encoding unit and the maximum two-divisionable luma encoding block. Therefore, by combining the information regarding the maximum size of the two-divisionable luma encoding block obtained from the bitstream with the information regarding the difference in luma block size, the size of the maximum luma encoding unit can be determined. Using the size of the maximum luma encoding unit, the size of the maximum chroma encoding unit can also be determined. For example, if the Y:Cb:Cr ratio according to the color format is 4:2:0, the size of the chroma block may be half the size of the luma block, and similarly, the size of the maximum chroma encoding unit may be half the size of the maximum luma encoding unit.

[0070] According to one embodiment, information regarding the maximum size of a binary splittable luminous encoding block is obtained from a bitstream, so the maximum size of the binary splittable luminous encoding block can be determined variably. Alternatively, the maximum size of a ternary splittable luminous encoding block can be fixed. For example, the maximum size of a ternary splittable luminous encoding block in image I may be 32x32, and the maximum size of a ternary splittable luminous encoding block in picture P or image B may be 64x64.

[0071] Additionally, the maximum encoding unit can be hierarchically divided into encoding units based on splitting mode information obtained from the bitstream. As splitting mode information, at least one of information indicating whether it is a quad split, information indicating whether it is a multi-split, splitting direction information, and splitting type information can be obtained from the bitstream.

[0072] For example, information indicating whether quad splitting is performed can indicate whether the current encoding unit will be quad split or not.

[0073] If the current encoding unit is not quad-splitting, information indicating multi-splitting can indicate whether the current encoding unit will no longer be split (NO_SPLIT) or whether it will be binary / ternary split.

[0074] If the current encoding unit is binary or binary split, the splitting direction information indicates that the current encoding unit is split in either the horizontal or vertical direction.

[0075] If the current encoding unit is split horizontally or vertically, the split type information indicates that the current encoding unit is split into binary or binary splits.

[0076] The splitting mode of the current encoding unit can be determined based on the splitting direction information and the splitting type information. The splitting mode when the current encoding unit is binary split in the horizontal direction can be determined as binary horizontal splitting (SPLIT_BT_HOR), when it is territorial split in the horizontal direction as territorial horizontal splitting (SPLIT_TT_HOR), when it is binary split in the vertical direction as binary vertical splitting (SPLIT_BT_VER), and when it is territorial split in the vertical direction as territorial vertical splitting (SPLIT_TT_VER).

[0077] The image decoding device (100) can obtain splitting mode information from a bitstream from a single empty string. The form of the bitstream received by the image decoding device (100) may include a fixed-length binary code, a unary code, a truncated unary code, a predetermined binary code, etc. The empty string represents information as a sequence of binary numbers. The empty string may consist of at least one bit. The image decoding device (100) can obtain splitting mode information corresponding to the empty string based on a splitting rule. Based on the single empty string, the image decoding device (100) can determine whether to quad split the encoding unit, whether not to split it, or the splitting direction and splitting type.

[0078] A coding unit may be smaller than or equal to a maximum coding unit. For example, since the maximum coding unit is a coding unit having the maximum size, it is also a coding unit. If the segmentation mode information for the maximum coding unit indicates that it is not segmented, the coding unit determined from the maximum coding unit has the same size as the maximum coding unit. If the segmentation mode information for the maximum coding unit indicates that it is segmented, the maximum coding unit may be segmented into coding units. Additionally, if the segmentation mode information for a coding unit indicates segmentation, the coding units may be segmented into coding units of smaller size. However, the segmentation of the image is not limited to this, and the maximum coding unit and the coding unit may not be distinguished. The segmentation of coding units is explained in more detail in FIGS. 3 through 16.

[0079] Additionally, one or more prediction blocks for prediction may be determined from the coding unit. The prediction blocks may be equal to or smaller than the coding unit. Additionally, one or more transformation blocks for transformation may be determined from the coding unit. The transformation blocks may be equal to or smaller than the coding unit. The transformation blocks may be referred to as the transformation unit.

[0080] The shape and size of the transformation block and the prediction block may not be related to each other.

[0081] In another embodiment, prediction can be performed using the encoding unit as a prediction block. Additionally, conversion can be performed using the encoding unit as a conversion block.

[0082] The division of the encoding unit is described in more detail in FIGS. 3 through 16. The current block and neighbor block of the present disclosure may represent one of the maximum encoding unit, the encoding unit, the prediction block, and the transformation block. Additionally, the current block or the current encoding unit is a block currently undergoing decoding or encoding, or a block currently undergoing division. The neighbor block may be a block restored prior to the current block. The neighbor block may be spatially or temporally adjacent to the current block. The neighbor block may be located on one of the lower-left, left, upper-left, upper, upper-right, right, or lower-right sides of the current block.

[0083] The above-described embodiment explains the operation related to the image decoding method performed by the image decoding device (100). Below, the operation of the image encoding device (200) that performs an image encoding method corresponding to the reverse process of the image decoding method will be explained through an embodiment of the present disclosure.

[0084] FIG. 2 illustrates a block diagram of an image encoding device (200) capable of encoding an image based on at least one of block shape information and segmented shape mode information according to one embodiment of the present disclosure.

[0085] The video encoding device (200) may include an encoding unit (220) and a bitstream generation unit (210). The encoding unit (220) may receive an input video and encode the input video. The encoding unit (220) may encode the input video to obtain at least one syntax element. The syntax element may include at least one of a skip flag, a prediction mode, a motion vector difference, a motion vector prediction method (or index), a transform quantized coefficient, a coded block pattern, a coded block flag, an intra prediction mode, a direct flag, a merge flag, a delta QP, a reference index, a prediction direction, and a transform index. The encoding unit (220) may determine a context model based on block shape information including at least one of the shape, direction, width, and height ratio or size of the encoding unit.

[0086] The bitstream generation unit (210) can generate a bitstream based on an encoded input image. For example, the bitstream generation unit (210) can generate a bitstream by entropy encoding syntax elements based on a context model. In one embodiment of the present disclosure, the image encoding device (100) can generate a bitstream according to an image encoding method to be described later. The image encoding device (100) can store the bitstream in a computer-readable storage medium.

[0087] In one embodiment of the present disclosure, the image encoding device (200) can transmit a bitstream to the image decoding device (100). For example, the image encoding device (200) can transmit a bitstream generated by an image encoding method.

[0088] According to one embodiment of the present disclosure, the encoding unit (220) of the image encoding device (200) can determine the shape of the encoding unit. For example, the encoding unit may be square or non-square in shape, and information indicating such shape may be included in block shape information.

[0089] According to one embodiment of the present disclosure, the encoding unit (220) may determine how the encoding unit will be divided. The encoding unit (220) may determine the form of at least one encoding unit included in the encoding unit, and the bitstream generation unit (210) may generate a bitstream including division form mode information including information about the form of such encoding unit.

[0090] According to one embodiment of the present disclosure, the encoding unit (220) may determine whether the encoding unit is divided or not. If the encoding unit (220) determines that only one encoding unit is included in the encoding unit or that the encoding unit is not divided, the bitstream generation unit (210) may generate a bitstream including division type mode information indicating that the encoding unit is not divided. Additionally, the encoding unit (220) may divide the encoding unit into a plurality of encoding units, and the bitstream generation unit (210) may generate a bitstream including division type mode information indicating that the encoding unit is divided into a plurality of encoding units.

[0091] According to one embodiment of the present disclosure, information indicating how many encoding units to divide or in which direction to divide may be included in the division type mode information. For example, the division type mode information may indicate dividing in at least one of the vertical direction and the horizontal direction, or not dividing.

[0092] The video encoding device (200) determines information regarding the segmentation mode based on the segmentation mode of the encoding unit. The video encoding device (200) determines a context model based on at least one of the ratio or size of the shape, direction, width, and height of the encoding unit. Then, the video encoding device (200) generates information regarding the segmentation mode for segmenting the encoding unit based on the context model as a bitstream.

[0093] The video encoding device (200) may obtain an array for matching at least one of the ratio or size of the shape, direction, width, and height of an encoding unit with an index for the context model in order to determine the context model. The video encoding device (200) may obtain an index for the context model based on at least one of the ratio or size of the shape, direction, width, and height of an encoding unit in the array. The video encoding device (200) may determine the context model based on the index for the context model.

[0094] The video encoding device (200) may determine a context model based further on block shape information including at least one of the ratio or size of the shape, direction, width, and height of a surrounding encoding unit adjacent to the encoding unit, in order to determine the context model. Additionally, the surrounding encoding unit may include at least one of the encoding units located on the lower left, left, upper left, upper, upper right, right, or lower right side of the encoding unit.

[0095] Additionally, the video encoding device (200) can compare the width of an upper peripheral encoding unit with the width of an encoding unit in order to determine a context model. Additionally, the video encoding device (200) can compare the height of left and right peripheral encoding units with the height of an encoding unit. Additionally, the video encoding device (200) can determine a context model based on the comparison results.

[0096] The operation of the video encoding device (200) includes content similar to the operation of the video decoding device (100) described in FIGS. 3 to 19, so a detailed description is omitted.

[0097] FIG. 3 illustrates a process in which an image decoding device (100) divides a current encoding unit to determine at least one encoding unit according to one embodiment of the present disclosure.

[0098] The block shape may include 4Nx4N, 4Nx2N, 2Nx4N, 4NxN, Nx4N, 32NxN, Nx32N, 16NxN, Nx16N, 8NxN, or Nx8N. Here, N may be a positive integer. Block shape information is information indicating at least one of the shape, orientation, width, and height ratio or size of the encoding unit.

[0099] The shape of the encoding unit may include square and non-square. When the width and height of the encoding unit are the same (i.e., when the block shape of the encoding unit is 4Nx4N), the image decoder (100) may determine the block shape information of the encoding unit as square. The image decoder (100) may determine the shape of the encoding unit as non-square.

[0100] When the width and height of the encoding unit are different (i.e., when the block shape of the encoding unit is 4Nx2N, 2Nx4N, 4NxN, Nx4N, 32NxN, Nx32N, 16NxN, Nx16N, 8NxN, or Nx8N), the image decoder (100) can determine the block shape information of the encoding unit as non-square. When the shape of the encoding unit is non-square, the image decoder (100) can determine the ratio of the width and height among the block shape information of the encoding unit as at least one of 1:2, 2:1, 1:4, 4:1, 1:8, 8:1, 1:16, 16:1, 1:32, or 32:1. Additionally, based on the width and height of the encoding unit, the image decoding device (100) can determine whether the encoding unit is in a horizontal direction or a vertical direction. Additionally, based on at least one of the width, height, or width of the encoding unit, the image decoding device (100) can determine the size of the encoding unit.

[0101] According to one embodiment of the present disclosure, the image decoding device (100) can determine the form of the encoding unit using block form information and can determine how the encoding unit is divided using division form mode information. That is, depending on what block form the block form information used by the image decoding device (100) represents, the method of dividing the encoding unit represented by the division form mode information can be determined.

[0102] The video decoder (100) can obtain split-form mode information from the bitstream. However, it is not limited thereto, and the video decoder (100) and the video encoding device (200) can determine pre-agreed split-form mode information based on block form information. The video decoder (100) can determine pre-agreed split-form mode information for a maximum encoding unit or a minimum encoding unit. For example, the video decoder (100) can determine the split-form mode information for the maximum encoding unit as quad split. Additionally, the video decoder (100) can determine the split-form mode information for the minimum encoding unit as "not split." Specifically, the video decoder (100) can determine the size of the maximum encoding unit to be 256x256. The video decoder (100) can determine the pre-agreed split-form mode information as quad split. Quad split is a split-form mode that divides both the width and height of the encoding unit into two equal parts. The image decoder (100) can obtain a 128x128 size encoding unit from a 256x256 size encoding unit based on the division mode information. Additionally, the image decoder (100) can determine the size of the minimum encoding unit to be 4x4. The image decoder (100) can obtain division mode information indicating "not divided" for the minimum encoding unit.

[0103] According to one embodiment of the present disclosure, the image decoding device (100) may use block shape information indicating that the current encoding unit is in the shape of a square. For example, the image decoding device (100) may determine whether to not divide the square encoding unit, to divide it vertically, to divide it horizontally, or to divide it into four encoding units, etc., according to the division shape mode information. Referring to FIG. 3, when the block shape information of the current encoding unit (300) indicates a square shape, the decoding unit (120) may not divide the encoding unit (310a) having the same size as the current encoding unit (300) according to the division shape mode information indicating that it is not divided, or may determine the divided encoding units (310b, 310c, 310d, 310e, 310f, etc.) based on the division shape mode information indicating a predetermined division method.

[0104] Referring to FIG. 3, the image decoding device (100) can determine two encoding units (310b) that divide the current encoding unit (300) in the vertical direction based on splitting form mode information indicating that it is divided in the vertical direction according to one embodiment of the present disclosure. The image decoding device (100) can determine two encoding units (310c) that divide the current encoding unit (300) in the horizontal direction based on splitting form mode information indicating that it is divided in the horizontal direction. The image decoding device (100) can determine four encoding units (310d) that divide the current encoding unit (300) in the vertical direction and the horizontal direction based on splitting form mode information indicating that it is divided in the vertical direction and the horizontal direction. The image decoding device (100) may determine three encoding units (310e) that divide the current encoding unit (300) in the vertical direction based on division form mode information indicating ternary division in the vertical direction according to one embodiment of the present disclosure. The image decoding device (100) may determine three encoding units (310f) that divide the current encoding unit (300) in the horizontal direction based on division form mode information indicating ternary division in the horizontal direction. However, the division form in which the square encoding unit can be divided should not be interpreted as being limited to the form described above, and may include various forms that the division form mode information can represent. The predetermined division forms in which the square encoding unit is divided will be specifically described below through one embodiment of the present disclosure.

[0105] FIG. 4 illustrates a process in which an image decoding device (100) determines at least one encoding unit by dividing a non-square encoding unit according to one embodiment of the present disclosure.

[0106] According to one embodiment of the present disclosure, the image decoding device (100) may use block shape information indicating that the current encoding unit is in a non-square shape. The image decoding device (100) may determine whether to not divide the current encoding unit of the non-square shape or to divide it in a predetermined way according to the division shape mode information. Referring to FIG. 4, when the block shape information of the current encoding unit (400 or 450) indicates a non-square shape, the image decoding device (100) may determine an encoding unit (410 or 460) having the same size as the current encoding unit (400 or 450) according to division shape mode information indicating that it is not divided, or determine divided encoding units (420a, 420b, 430a, 430b, 430c, 470a, 470b, 480a, 480b, 480c) based on division shape mode information indicating a predetermined division method. A predetermined division method in which a non-square encoding unit is divided will be specifically described below through an embodiment of the present disclosure.

[0107] According to one embodiment of the present disclosure, the image decoding device (100) can determine the form in which the encoding unit is divided using the division form mode information, and in this case, the division form mode information may indicate the number of at least one encoding unit generated by dividing the encoding unit. Referring to FIG. 4, when the division form mode information indicates that the current encoding unit (400 or 450) is divided into two encoding units, the image decoding device (100) can determine two encoding units (420a, 420b, or 470a, 470b) included in the current encoding unit by dividing the current encoding unit (400 or 450) based on the division form mode information.

[0108] According to one embodiment of the present disclosure, when an image decoding device (100) divides a current encoding unit (400 or 450) in a non-square shape based on division shape mode information, the image decoding device (100) may divide the current encoding unit by considering the position of the long side of the current encoding unit (400 or 450) in a non-square shape. For example, the image decoding device (100) may determine a plurality of encoding units by dividing the current encoding unit (400 or 450) in a direction that divides the long side of the current encoding unit (400 or 450) by considering the shape of the current encoding unit (400 or 450).

[0109] According to one embodiment of the present disclosure, when the segmentation mode information indicates that the encoding unit is divided into an odd number of blocks (terminal segmentation), the image decoder (100) can determine an odd number of encoding units included in the current encoding unit (400 or 450). For example, when the segmentation mode information indicates that the current encoding unit (400 or 450) is divided into three encoding units, the image decoder (100) can divide the current encoding unit (400 or 450) into three encoding units (430a, 430b, 430c, 480a, 480b, 480c).

[0110] According to one embodiment of the present disclosure, the ratio of the width to the height of the current encoding unit (400 or 450) may be 4:1 or 1:4. When the ratio of the width to the height is 4:1, the block shape information may be in the horizontal direction because the width is longer than the height. When the ratio of the width to the height is 1:4, the block shape information may be in the vertical direction because the width is shorter than the height. The image decoder (100) may determine to divide the current encoding unit into an odd number of blocks based on the division shape mode information. Additionally, the image decoder (100) may determine the division direction of the current encoding unit (400 or 450) based on the block shape information of the current encoding unit (400 or 450). For example, if the current encoding unit (400) is in a vertical direction, the image decoding device (100) can determine the encoding units (430a, 430b, 430c) by dividing the current encoding unit (400) in a horizontal direction. Also, if the current encoding unit (450) is in a horizontal direction, the image decoding device (100) can determine the encoding units (480a, 480b, 480c) by dividing the current encoding unit (450) in a vertical direction.

[0111] According to one embodiment of the present disclosure, an image decoding device (100) may determine an odd number of encoding units included in a current encoding unit (400 or 450), and the sizes of the determined encoding units may not all be the same. For example, among the determined odd number of encoding units (430a, 430b, 430c, 480a, 480b, 480c), the size of a certain encoding unit (430b or 480b) may have a different size from the other encoding units (430a, 430c, 480a, 480c). That is, the current encoding unit (400 or 450) can be divided and determined as a encoding unit, and the encoding unit can have multiple types of sizes, and in some cases, an odd number of encoding units (430a, 430b, 430c, 480a, 480b, 480c) may each have different sizes.

[0112] According to one embodiment of the present disclosure, when the segmentation mode information indicates that the encoding unit is divided into an odd number of blocks, the image decoding device (100) can determine the odd number of encoding units included in the current encoding unit (400 or 450), and furthermore, the image decoding device (100) can impose a predetermined limit on at least one encoding unit among the odd number of encoding units generated by the segmentation. Referring to FIG. 4, the image decoding device (100) can perform the decoding process for the central encoding unit (430b, 480b) among the three encoding units (430a, 430b, 430c, 480a, 480b, 480c) generated by the segmentation of the current encoding unit (400 or 450) differently from the other encoding units (430a, 430c, 480a, 480c). For example, the video decoding device (100) may restrict the centrally located encoding unit (430b, 480b) from being further divided unlike other encoding units (430a, 430c, 480a, 480c), or restrict it to being divided only a predetermined number of times.

[0113] FIG. 5 illustrates a process in which an image decoding device (100) divides a encoding unit based on at least one of block form information and division form mode information according to one embodiment of the present disclosure.

[0114] According to one embodiment of the present disclosure, an image decoding device (100) may determine whether to divide a square-shaped first encoding unit (500) into encoding units or not to divide it based on at least one of block shape information and division shape mode information. According to one embodiment of the present disclosure, if the division shape mode information indicates that the first encoding unit (500) is divided in a horizontal direction, the image decoding device (100) may divide the first encoding unit (500) in a horizontal direction to determine a second encoding unit (510). The first encoding unit, the second encoding unit, and the third encoding unit used according to one embodiment of the present disclosure are terms used to understand the relationship before and after division between the encoding units. For example, if the first encoding unit is divided, the second encoding unit may be determined, and if the second encoding unit is divided, the third encoding unit may be determined. In the following, the relationship between the first encoding unit, the second encoding unit, and the third encoding unit used may be understood as following the features described above.

[0115] According to one embodiment of the present disclosure, the image decoding device (100) may determine whether to divide the determined second encoding unit (510) into encoding units or not to divide it based on the division shape mode information. Referring to FIG. 5, the image decoding device (100) may divide the determined non-square second encoding unit (510) into at least one third encoding unit (520a, 520b, 520c, 520d, etc.) by dividing the first encoding unit (500) based on the division shape mode information, or may not divide the second encoding unit (510). The image decoding device (100) can obtain split-form mode information, and the image decoding device (100) can divide a first encoding unit (500) based on the obtained split-form mode information to divide a plurality of second encoding units (e.g., 510) of various forms, and the second encoding units (510) can be divided according to the method in which the first encoding unit (500) was divided based on the split-form mode information. According to one embodiment of the present disclosure, when the first encoding unit (500) is divided into a second encoding unit (510) based on the split-form mode information for the first encoding unit (500), the second encoding unit (510) can also be divided into a third encoding unit (e.g., 520a, 520b, 520c, 520d, etc.) based on the split-form mode information for the second encoding unit (510). That is, the encoding unit can be recursively partitioned based on partitioning mode information associated with each encoding unit. Thus, a square encoding unit can be determined from a non-square encoding unit, and a non-square encoding unit can be determined by recursively partitioning this square encoding unit.

[0116] Referring to FIG. 5, among the odd number of third encoding units (520b, 520c, 520d) determined by dividing a second encoding unit (510) of a non-square shape, a predetermined encoding unit (e.g., a central encoding unit or a square encoding unit) may be recursively divided. According to one embodiment of the present disclosure, a third encoding unit (520b) of a non-square shape, which is one of the odd number of third encoding units (520b, 520c, 520d), may be divided horizontally into a plurality of fourth encoding units. A fourth encoding unit (530b or 530d) of a non-square shape, which is one of the plurality of fourth encoding units (530a, 530b, 530c, 530d), may again be divided into a plurality of encoding units. For example, a non-square fourth encoding unit (530b or 530d) may be further subdivided into an odd number of encoding units. A method that can be used for the recursive subdivision of encoding units will be described later through an embodiment of the present disclosure.

[0117] According to one embodiment of the present disclosure, the image decoding device (100) may divide each of the third encoding units (520a, 520b, 520c, 520d, etc.) into encoding units based on the division shape mode information. Additionally, the image decoding device (100) may decide not to divide the second encoding unit (510) based on the division shape mode information. According to one embodiment of the present disclosure, the image decoding device (100) may divide the non-square second encoding unit (510) into an odd number of third encoding units (520b, 520c, 520d). The image decoding device (100) may place a certain limit on a certain third encoding unit among the odd number of third encoding units (520b, 520c, 520d). For example, the video decoding device (100) may limit the encoding unit (520c) located in the middle of the odd number of third encoding units (520b, 520c, 520d) so that it is not further divided or is limited to being divided a set number of times.

[0118] Referring to FIG. 5, the image decoding device (100) may limit the middle encoding unit (520c) among the odd number of third encoding units (520b, 520c, 520d) included in the second encoding unit (510) of a non-square shape to no longer be divided, to be divided into a predetermined division form (e.g., divided into only 4 encoding units or divided into a form corresponding to the divided form of the second encoding unit (510)), or to be divided only a predetermined number of times (e.g., divided only n times, n > 0). However, the above limitation on the middle encoding unit (520c) is merely a simple example and should not be interpreted as being limited to the above-described examples, but should be interpreted as including various limitations that allow the middle encoding unit (520c) to be decoded differently from the other encoding units (520b, 520d).

[0119] According to one embodiment of the present disclosure, an image decoding device (100) can obtain splitting form mode information used to split a current encoding unit at a predetermined location within the current encoding unit.

[0120] FIG. 6 illustrates a method for an image decoding device (100) to determine a predetermined encoding unit among an odd number of encoding units according to one embodiment of the present disclosure.

[0121] Referring to FIG. 6, the segmentation mode information of the current encoding unit (600, 650) can be obtained from a sample at a specific location among a plurality of samples included in the current encoding unit (600, 650) (e.g., a sample located in the center (640, 690)). However, the specific location within the current encoding unit (600) where at least one of such segmentation mode information can be obtained should not be interpreted as being limited to the center location shown in FIG. 6, and should be interpreted as including various locations within the current encoding unit (600) (e.g., top, bottom, left, right, top-left, bottom-left, top-right, or bottom-right, etc.). The image decoding device (100) can obtain the segmentation mode information obtained from the specific location and decide whether to divide the current encoding unit into encoding units of various shapes and sizes or not to divide it.

[0122] According to one embodiment of the present disclosure, the image decoding device (100) may select one of the encoding units when the current encoding unit is divided into a predetermined number of encoding units. There may be various methods for selecting one of the plurality of encoding units, and such methods will be described later through one embodiment of the present disclosure.

[0123] According to one embodiment of the present disclosure, the image decoding device (100) can divide the current encoding unit into a plurality of encoding units and determine the encoding unit at a predetermined position.

[0124] According to one embodiment of the present disclosure, an image decoding device (100) may use information indicating the position of each of the odd number of encoding units to determine the encoding unit located in the middle among the odd number of encoding units. Referring to FIG. 6, the image decoding device (100) may divide the current encoding unit (600) or the current encoding unit (650) to determine the odd number of encoding units (620a, 620b, 620c) or the odd number of encoding units (660a, 660b, 660c). The image decoding device (100) may determine the middle encoding unit (620b) or the middle encoding unit (660b) by using information regarding the positions of the odd number of encoding units (620a, 620b, 620c) or the odd number of encoding units (660a, 660b, 660c). For example, the image decoding device (100) can determine the centrally located encoding unit (620b) by determining the positions of the encoding units (620a, 620b, 620c) based on information indicating the positions of a predetermined sample included in the encoding units (620a, 620b, 620c). Specifically, the image decoding device (100) can determine the centrally located encoding unit (620b) by determining the positions of the encoding units (620a, 620b, 620c) based on information indicating the positions of the upper-left samples (630a, 630b, 630c) of the encoding units (620a, 620b, 620c).

[0125] According to one embodiment of the present disclosure, information indicating the location of the upper-left sample (630a, 630b, 630c) included in each of the encoding units (620a, 620b, 620c) may include information regarding the location or coordinates within the picture of the encoding units (620a, 620b, 620c). According to one embodiment of the present disclosure, information indicating the location of the upper-left sample (630a, 630b, 630c) included in each of the encoding units (620a, 620b, 620c) may include information indicating the width or height of the encoding units (620a, 620b, 620c) included in the current encoding unit (600), and such width or height may correspond to information indicating the difference between coordinates within the picture of the encoding units (620a, 620b, 620c). That is, the image decoding device (100) can determine the centrally located encoding unit (620b) by directly using information about the position or coordinates of the encoding units (620a, 620b, 620c) within the picture, or by using information about the width or height of the encoding unit corresponding to the difference value between the coordinates.

[0126] According to one embodiment of the present disclosure, information indicating the location of a sample (630a) at the upper left of an upper encoding unit (620a) may be represented by the (xa, ya) coordinates, information indicating the location of a sample (530b) at the upper left of a middle encoding unit (620b) may be represented by the (xb, yb) coordinates, and information indicating the location of a sample (630c) at the upper left of a lower encoding unit (620c) may be represented by the (xc, yc) coordinates. An image decoding device (100) may determine the middle encoding unit (620b) using the coordinates of the upper left samples (630a, 630b, 630c) included in each of the encoding units (620a, 620b, 620c). For example, when the coordinates of the upper-left samples (630a, 630b, 630c) are sorted in ascending or descending order, the encoding unit (620b) containing the coordinates (xb, yb) of the sample (630b) located in the middle can be determined as the encoding unit located in the middle among the encoding units (620a, 620b, 620c) determined by dividing the current encoding unit (600). However, the coordinates indicating the position of the upper-left samples (630a, 630b, 630c) may represent absolute positions within the picture, and furthermore, based on the position of the upper-left sample (630a) of the upper-left of the upper-left of the middle encoding unit (620b), the (dxb, dyb) coordinates, which represent the relative position of the upper-left sample (630b) of the middle encoding unit (620b), and the (dxc, dyc) coordinates, which represent the relative position of the upper-left sample (630c) of the lower encoding unit (620c) may also be used. In addition, the method of determining the encoding unit of a predetermined position by using the coordinates of the corresponding sample as information indicating the position of the sample included in the encoding unit should not be interpreted as being limited to the method described above, but should be interpreted as various arithmetic methods that can utilize the coordinates of the sample.

[0127] According to one embodiment of the present disclosure, the image decoding device (100) can divide the current encoding unit (600) into a plurality of encoding units (620a, 620b, 620c) and select an encoding unit among the encoding units (620a, 620b, 620c) according to a predetermined criterion. For example, the image decoding device (100) can select an encoding unit (620b) of a different size among the encoding units (620a, 620b, 620c).

[0128] According to one embodiment of the present disclosure, an image decoding device (100) can determine the width or height of each of the encoding units (620a, 620b, 620c) using the (xa, ya) coordinates, which are information indicating the location of a sample (630a) at the top left of the upper encoding unit (620a); the (xb, yb) coordinates, which are information indicating the location of a sample (630b) at the top left of the middle encoding unit (620b); and the (xc, yc) coordinates, which are information indicating the location of a sample (630c) at the top left of the lower encoding unit (620c). The image decoding device (100) can determine the size of each of the encoding units (620a, 620b, 620c) using coordinates (xa, ya), (xb, yb), and (xc, yc) which represent the positions of the encoding units (620a, 620b, 620c). According to one embodiment of the present disclosure, the image decoding device (100) can determine the width of the upper encoding unit (620a) as the width of the current encoding unit (600). The image decoding device (100) can determine the height of the upper encoding unit (620a) as yb-ya. According to one embodiment of the present disclosure, the image decoding device (100) can determine the width of the middle encoding unit (620b) as the width of the current encoding unit (600). The image decoding device (100) can determine the height of the middle encoding unit (620b) as yc-yb. According to one embodiment of the present disclosure, the image decoding device (100) can determine the width or height of the lower encoding unit using the width or height of the current encoding unit and the width and height of the upper encoding unit (620a) and the middle encoding unit (620b). The image decoding device (100) can determine an encoding unit having a different size from other encoding units based on the width and height of the determined encoding units (620a, 620b, 620c).Referring to FIG. 6, the image decoding device (100) can determine a middle encoding unit (620b) having a size different from that of the upper encoding unit (620a) and the lower encoding unit (620c) as the encoding unit of a predetermined position. However, since the process of the image decoding device (100) described above determining the encoding unit having a size different from other encoding units is merely one embodiment of determining the encoding unit of a predetermined position using the size of the encoding unit determined based on sample coordinates, various processes of determining the encoding unit of a predetermined position by comparing the size of the encoding unit determined according to the predetermined sample coordinates may be used.

[0129] The image decoding device (100) can determine the width or height of each of the encoding units (660a, 660b, 660c) using the (xd, yd) coordinates, which are information indicating the location of the upper-left sample (670a) of the left encoding unit (660a), the (xe, ye) coordinates, which are information indicating the location of the upper-left sample (670b) of the middle encoding unit (660b), and the (xf, yf) coordinates, which are information indicating the location of the upper-left sample (670c) of the right encoding unit (660c). The image decoding device (100) can determine the size of each of the encoding units (660a, 660b, 660c) using the (xd, yd), (xe, ye), and (xf, yf) coordinates, which are information indicating the location of the encoding units (660a, 660b, 660c).

[0130] According to one embodiment of the present disclosure, the image decoding device (100) may determine the width of the left encoding unit (660a) as xe-xd. The image decoding device (100) may determine the height of the left encoding unit (660a) as the height of the current encoding unit (650). According to one embodiment of the present disclosure, the image decoding device (100) may determine the width of the middle encoding unit (660b) as xf-xe. The image decoding device (100) may determine the height of the middle encoding unit (660b) as the height of the current encoding unit (600). According to one embodiment of the present disclosure, the image decoding device (100) may determine the width or height of the right encoding unit (660c) using the width or height of the current encoding unit (650) and the width and height of the left encoding unit (660a) and the middle encoding unit (660b). The image decoding device (100) can determine a encoding unit having a different size from other encoding units based on the width and height of the determined encoding units (660a, 660b, 660c). Referring to FIG. 6, the image decoding device (100) can determine a middle encoding unit (660b) having a different size from the left encoding unit (660a) and the right encoding unit (660c) as the encoding unit of a predetermined position. However, since the process of the image decoding device (100) described above determining a encoding unit having a different size from other encoding units is merely one embodiment of determining the encoding unit of a predetermined position using the size of the encoding unit determined based on sample coordinates, various processes of determining the encoding unit of a predetermined position by comparing the size of the encoding unit determined according to the predetermined sample coordinates may be used.

[0131] However, the sample location considered to determine the location of the encoding unit should not be interpreted as being limited to the upper left corner described above, and can be interpreted as allowing the use of information regarding the location of any sample included in the encoding unit.

[0132] According to one embodiment of the present disclosure, the image decoding device (100) may select an encoding unit at a predetermined position among an odd number of encoding units determined by dividing the current encoding unit, taking into consideration the shape of the current encoding unit. For example, if the current encoding unit is a non-square shape where the width is longer than the height, the image decoding device (100) may determine an encoding unit at a predetermined position according to the horizontal direction. That is, the image decoding device (100) may determine one of the encoding units at a different position in the horizontal direction and impose a restriction on that encoding unit. If the current encoding unit is a non-square shape where the height is longer than the width, the image decoding device (100) may determine an encoding unit at a predetermined position according to the vertical direction. That is, the image decoding device (100) may determine one of the encoding units at a different position in the vertical direction and impose a restriction on that encoding unit.

[0133] According to one embodiment of the present disclosure, an image decoding device (100) may use information indicating the location of each of an even number of encoding units to determine an encoding unit at a predetermined location among an even number of encoding units. The image decoding device (100) may determine an even number of encoding units by dividing (binary division) the current encoding unit and may determine an encoding unit at a predetermined location using information regarding the locations of the even number of encoding units. A specific process for this may be omitted as it may correspond to the process of determining an encoding unit at a predetermined location (e.g., a middle location) among an odd number of encoding units described above in FIG. 6.

[0134] According to one embodiment of the present disclosure, when a current encoding unit of a non-square shape is divided into a plurality of encoding units, certain information regarding the encoding unit at a certain position may be used during the division process to determine the encoding unit at a certain position among the plurality of encoding units. For example, an image decoding device (100) may use at least one of block shape information and division shape mode information stored in a sample included in the middle encoding unit during the division process to determine the encoding unit located in the middle among the encoding units into which the current encoding unit is divided into a plurality of encoding units.

[0135] Referring to FIG. 6, the image decoding device (100) can divide the current encoding unit (600) into a plurality of encoding units (620a, 620b, 620c) based on the division form mode information, and can determine the encoding unit (620b) located in the middle among the plurality of encoding units (620a, 620b, 620c). Furthermore, the image decoding device (100) can determine the encoding unit (620b) located in the middle by considering the location where the division form mode information is obtained. That is, the segmentation mode information of the current encoding unit (600) can be obtained from a sample (640) located in the middle of the current encoding unit (600), and based on the segmentation mode information, if the current encoding unit (600) is divided into a plurality of encoding units (620a, 620b, 620c), the encoding unit (620b) containing the sample (640) can be determined as the encoding unit located in the middle. However, the information used to determine the encoding unit located in the middle should not be interpreted as being limited to segmentation mode information, and various types of information may be used in the process of determining the encoding unit located in the middle.

[0136] According to one embodiment of the present disclosure, a predetermined information for identifying a coding unit at a predetermined location may be obtained from a predetermined sample included in the coding unit to be determined. Referring to FIG. 6, an image decoding device (100) may use a segmentation mode information obtained from a sample at a predetermined location within the current coding unit (600) (for example, a sample at the center of the current coding unit (600)) to determine a coding unit at a predetermined location (for example, a coding unit located in the center of the multiple divided coding units) among a plurality of coding units (620a, 620b, 620c) determined by dividing the current coding unit (600). That is, the image decoding device (100) can determine a sample at a predetermined location by considering the block shape of the current encoding unit (600), and the image decoding device (100) can determine a encoding unit (620b) containing a sample from which certain information (e.g., division shape mode information) can be obtained among a plurality of encoding units (620a, 620b, 620c) from which the current encoding unit (600) is divided and determined, and can impose a certain limit. Referring to FIG. 6, according to one embodiment of the present disclosure, the image decoding device (100) can determine a sample (640) located in the middle of the current encoding unit (600) as a sample from which certain information can be obtained, and the image decoding device (100) can impose a certain limit on the encoding unit (620b) containing such a sample (640) during the decoding process. However, the location of the sample from which the specified information can be obtained should not be interpreted as being limited to the location described above, but can be interpreted as samples at any location included in the encoding unit (620b) to be determined for the purpose of imposing a limit.

[0137] According to one embodiment of the present disclosure, the location of a sample from which a predetermined information can be obtained may be determined according to the shape of the current encoding unit (600). According to one embodiment of the present disclosure, block shape information may determine whether the shape of the current encoding unit is square or non-square, and may determine the location of a sample from which a predetermined information can be obtained according to the shape. For example, the image decoding device (100) may determine a sample located on a boundary that divides at least one of the width and height of the current encoding unit in half using at least one of the information regarding the width and height of the current encoding unit as a sample from which a predetermined information can be obtained. As another example, if the block shape information related to the current encoding unit indicates that it is a non-square shape, the image decoding device (100) may determine one of the samples adjacent to the boundary that divides the long side of the current encoding unit in half as a sample from which a predetermined information can be obtained.

[0138] According to one embodiment of the present disclosure, when the image decoding device (100) divides the current encoding unit into a plurality of encoding units, it may use division form mode information to determine the encoding unit at a predetermined position among the plurality of encoding units. According to one embodiment of the present disclosure, the image decoding device (100) may obtain division form mode information from a sample at a predetermined position included in the encoding unit, and the image decoding device (100) may divide the plurality of encoding units generated by dividing the current encoding unit using the division form mode information obtained from a sample at a predetermined position included in each of the plurality of encoding units. That is, the encoding unit may be recursively divided using the division form mode information obtained from a sample at a predetermined position included in each of the encoding units. Since the recursive division process of the encoding unit has been described in detail through FIG. 5, a detailed explanation will be omitted.

[0139] According to one embodiment of the present disclosure, the image decoding device (100) can determine at least one encoding unit by dividing the current encoding unit, and can determine the order in which the at least one encoding unit is decoded according to a predetermined block (e.g., the current encoding unit).

[0140] FIG. 7 illustrates the order in which a plurality of encoding units are processed when an image decoding device (100) determines a plurality of encoding units by dividing a current encoding unit according to one embodiment of the present disclosure.

[0141] According to one embodiment of the present disclosure, the image decoding device (100) may determine a second encoding unit (710a, 710b) by dividing a first encoding unit (700) in a vertical direction according to the splitting form mode information, determine a second encoding unit (730a, 730b) by dividing the first encoding unit (700) in a horizontal direction, or determine a second encoding unit (750a, 750b, 750c, 750d) by dividing the first encoding unit (700) in both a vertical and a horizontal direction.

[0142] Referring to FIG. 7, the image decoding device (100) can determine the order of processing the determined second encoding units (710a, 710b) in the horizontal direction (710c) by dividing the first encoding unit (700) in the vertical direction. The image decoding device (100) can determine the processing order of the determined second encoding units (730a, 730b) in the vertical direction (730c) by dividing the first encoding unit (700) in the horizontal direction. The image decoding device (100) can determine the second encoding unit (750a, 750b, 750c, 750d) determined by dividing the first encoding unit (700) into vertical and horizontal directions, according to a predetermined order in which encoding units located in one row are processed and then encoding units located in the next row are processed (e.g., raster scan order or z scan order (750e), etc.).

[0143] According to one embodiment of the present disclosure, the image decoding device (100) can recursively divide the encoding units. Referring to FIG. 7, the image decoding device (100) can divide the first encoding unit (700) to determine a plurality of encoding units (710a, 710b, 730a, 730b, 750a, 750b, 750c, 750d), and can recursively divide each of the determined plurality of encoding units (710a, 710b, 730a, 730b, 750a, 750b, 750c, 750d). A method of dividing multiple encoding units (710a, 710b, 730a, 730b, 750a, 750b, 750c, 750d) may be a method corresponding to a method of dividing the first encoding unit (700). Accordingly, the multiple encoding units (710a, 710b, 730a, 730b, 750a, 750b, 750c, 750d) may each be independently divided into multiple encoding units. Referring to FIG. 7, the image decoding device (100) may determine the second encoding units (710a, 710b) by dividing the first encoding unit (700) in a vertical direction, and furthermore, may determine whether to independently divide or not divide each of the second encoding units (710a, 710b).

[0144] According to one embodiment of the present disclosure, the image decoding device (100) may divide the second encoding unit (710a) on the left side horizontally into third encoding units (720a, 720b), and may not divide the second encoding unit (710b) on the right side.

[0145] According to one embodiment of the present disclosure, the processing order of the encoding units may be determined based on the process of dividing the encoding units. In other words, the processing order of the divided encoding units may be determined based on the processing order of the encoding units immediately before they are divided. The image decoding device (100) may determine the processing order of the third encoding units (720a, 720b), which are determined by dividing the second encoding unit (710a) on the left, independently of the second encoding unit (710b) on the right. Since the third encoding units (720a, 720b) ​​are determined by dividing the second encoding unit (710a) on the left in a horizontal direction, the third encoding units (720a, 720b) ​​may be processed in a vertical direction (720c). In addition, since the processing order of the second encoding unit (710a) on the left and the second encoding unit (710b) on the right corresponds to the horizontal direction (710c), the third encoding unit (720a, 720b) ​​included in the second encoding unit (710a) on the left can be processed in the vertical direction (720c) before the right encoding unit (710b) is processed. The above description is intended to explain the process in which the processing order of the encoding units is determined according to the encoding unit before division, and therefore should not be interpreted as being limited to the above-described embodiment, but should be interpreted as being used in various ways in which encoding units determined by division in various forms can be processed independently according to a predetermined order.

[0146] FIG. 8 illustrates a process in which, according to one embodiment of the present disclosure, an image decoding device (100) determines that the current encoding unit is divided into an odd number of encoding units when the encoding unit cannot be processed in a predetermined order.

[0147] According to one embodiment of the present disclosure, the image decoding device (100) may determine that the current encoding unit is divided into an odd number of encoding units based on acquired segmentation mode information. Referring to FIG. 8, a square-shaped first encoding unit (800) may be divided into non-square-shaped second encoding units (810a, 810b), and the second encoding units (810a, 810b) may each be independently divided into third encoding units (820a, 820b, 820c, 820d, 820e). According to one embodiment of the present disclosure, the image decoding device (100) can determine a plurality of third encoding units (820a, 820b) by dividing the left encoding unit (810a) among the second encoding units in a horizontal direction, and the right encoding unit (810b) can be divided into an odd number of third encoding units (820c, 820d, 820e).

[0148] According to one embodiment of the present disclosure, the image decoding device (100) can determine whether there exists an odd number of divided encoding units by determining whether the third encoding units (820a, 820b, 820c, 820d, 820e) can be processed in a predetermined order. Referring to FIG. 8, the image decoding device (100) can determine the third encoding units (820a, 820b, 820c, 820d, 820e) by recursively dividing the first encoding unit (800). The video decoding device (100) can determine whether the first encoding unit (800), the second encoding unit (810a, 810b), or the third encoding unit (820a, 820b, 820c, 820d, 820e) are divided into an odd number of encoding units based on at least one of block form information and division form mode information. For example, the encoding unit located on the right among the second encoding units (810a, 810b) may be divided into an odd number of third encoding units (820c, 820d, 820e). The order in which a plurality of encoding units included in the first encoding unit (800) are processed may be a predetermined order (e.g., z-scan order (830)), and the image decoding device (100) may determine whether the right second encoding unit (810b) is divided into an odd number of determined third encoding units (820c, 820d, 820e) can be processed according to the predetermined order.

[0149] According to one embodiment of the present disclosure, an image decoding device (100) can determine whether a third encoding unit (820a, 820b, 820c, 820d, 820e) included in a first encoding unit (800) satisfies a condition that the third encoding unit (820a, 820b, 820c, 820d, 820e) can be processed in a predetermined order, and the condition relates to whether at least one of the width and height of the second encoding unit (810a, 810b) is divided in half according to the boundary of the third encoding unit (820a, 820b, 820c, 820d, 820e). For example, the third encoding unit (820a, 820b) determined by dividing the height of the left second encoding unit (810a) in a non-square shape in half may satisfy the condition. Since the boundaries of the third encoding units (820c, 820d, 820e), which are determined by dividing the right second encoding unit (810b) into three encoding units, do not divide the width or height of the right second encoding unit (810b) in half, the third encoding units (820c, 820d, 820e) may be determined not to satisfy the condition. In the case of such non-satisfaction of the condition, the image decoding device (100) determines that there is a disconnection in the scan order, and based on the result of the determination, the right second encoding unit (810b) may be determined to be divided into an odd number of encoding units. According to one embodiment of the present disclosure, when the image decoding device (100) is divided into an odd number of encoding units, it may place a certain restriction on the encoding unit at a certain position among the divided encoding units, and since the details of such restriction or the certain position, etc. have been described in detail through one embodiment of the present disclosure, a detailed explanation will be omitted.

[0150] FIG. 9 illustrates a process in which an image decoding device (100) divides a first encoding unit (900) to determine at least one encoding unit according to one embodiment of the present disclosure.

[0151] According to one embodiment of the present disclosure, an image decoding device (100) may divide a first encoding unit (900) based on division shape mode information obtained through a bitstream acquisition unit (110). A square-shaped first encoding unit (900) may be divided into four square-shaped encoding units or into a plurality of non-square-shaped encoding units. For example, referring to FIG. 9, if the first encoding unit (900) is square and the division shape mode information indicates that it is divided into non-square encoding units, the image decoding device (100) may divide the first encoding unit (900) into a plurality of non-square encoding units. Specifically, when the splitting mode information indicates that the first encoding unit (900) is divided in a horizontal or vertical direction to determine an odd number of encoding units, the image decoding device (100) can divide the square-shaped first encoding unit (900) into an odd number of encoding units, such as a second encoding unit (910a, 910b, 910c) determined by dividing in a vertical direction or a second encoding unit (920a, 920b, 920c) determined by dividing in a horizontal direction.

[0152] According to one embodiment of the present disclosure, an image decoding device (100) can determine whether a second encoding unit (910a, 910b, 910c, 920a, 920b, 920c) included in a first encoding unit (900) satisfies a condition that the second encoding unit (910a, 910b, 910c, 920a, 920b, 920c) can be processed in a predetermined order, and the condition relates to whether at least one of the width and height of the first encoding unit (900) is divided in half according to the boundary of the second encoding unit (910a, 910b, 910c, 920a, 920b, 920c). Referring to FIG. 9, the boundaries of the second encoding units (910a, 910b, 910c), which are determined by dividing the square-shaped first encoding unit (900) in the vertical direction, do not divide the width of the first encoding unit (900) in half, so the first encoding unit (900) may be determined not to satisfy the condition of being processed in a predetermined order. Additionally, the boundaries of the second encoding units (920a, 920b, 920c), which are determined by dividing the square-shaped first encoding unit (900) in the horizontal direction, do not divide the height of the first encoding unit (900) in half, so the first encoding unit (900) may be determined not to satisfy the condition of being processed in a predetermined order. The image decoding device (100) determines that if these conditions are not satisfied, there is a disconnection in the scan order, and based on the result of the determination, the first encoding unit (900) may be divided into an odd number of encoding units. According to one embodiment of the present disclosure, when the image decoding device (100) is divided into an odd number of encoding units, it may place a certain restriction on the encoding unit at a certain position among the divided encoding units. Since the details of such restriction or the certain position, etc., have been described in detail through one embodiment of the present disclosure, a detailed explanation will be omitted.

[0153] According to one embodiment of the present disclosure, the image decoding device (100) can divide the first encoding unit to determine various forms of encoding units.

[0154] Referring to FIG. 9, the image decoding device (100) can divide a square-shaped first encoding unit (900) and a non-square-shaped first encoding unit (930 or 950) into various types of encoding units.

[0155] FIG. 10 illustrates that, according to one embodiment of the present disclosure, when a second encoding unit in a non-square shape determined by dividing a first encoding unit (1000) satisfies a predetermined condition, the shape in which the second encoding unit can be divided is limited.

[0156] According to one embodiment of the present disclosure, an image decoding device (100) may decide to divide a square-shaped first encoding unit (1000) into non-square-shaped second encoding units (1010a, 1010b, 1020a, 1020b) based on division shape mode information obtained through a bitstream acquisition unit (110). The second encoding units (1010a, 1010b, 1020a, 1020b) may be divided independently. Accordingly, the image decoding device (100) may decide to divide into a plurality of encoding units or not divide based on division shape mode information related to each of the second encoding units (1010a, 1010b, 1020a, 1020b). According to one embodiment of the present disclosure, the image decoding device (100) may determine a third encoding unit (1012a, 1012b) by dividing a left second encoding unit (1010a), which is a non-square shape determined by dividing a first encoding unit (1000) in a vertical direction, in a horizontal direction. However, when the image decoding device (100) divides the left second encoding unit (1010a) in a horizontal direction, the right second encoding unit (1010b) may be restricted so that it cannot be divided in the same horizontal direction as the left second encoding unit (1010a) was divided. If the right second encoding unit (1010b) is divided in the same direction to determine the third encoding unit (1014a, 1014b), the left second encoding unit (1010a) and the right second encoding unit (1010b) may be divided independently in the horizontal direction to determine the third encoding unit (1012a, 1012b, 1014a, 1014b). However, this is the same result as the image decoding device (100) dividing the first encoding unit (1000) into four square-shaped second encoding units (1030a, 1030b, 1030c, 1030d) based on the division shape mode information, and this may be inefficient in terms of image decoding.

[0157] According to one embodiment of the present disclosure, the image decoding device (100) may determine a third encoding unit (1022a, 1022b, 1024a, 1024b) by dividing a first encoding unit (1000) in a horizontal direction and a second encoding unit (1020a or 1020b) in a non-square shape in a vertical direction. However, if the image decoding device (100) divides one of the second encoding units (e.g., the upper second encoding unit (1020a)) in a vertical direction, it may restrict the other second encoding unit (e.g., the lower encoding unit (1020b)) from being divided in the same vertical direction as the upper second encoding unit (1020a) for the reasons described above.

[0158] FIG. 11 illustrates the process of a video decoder (100) dividing square-shaped encoding units when the divided shape mode information cannot be divided into four square-shaped encoding units according to one embodiment of the present disclosure.

[0159] According to one embodiment of the present disclosure, an image decoding device (100) can determine a second encoding unit (1110a, 1110b, 1120a, 1120b, etc.) by dividing a first encoding unit (1100) based on division form mode information. The division form mode information may include information on various forms in which the encoding unit can be divided, but the information on various forms may not include information for dividing into four square-shaped encoding units. According to such division form mode information, the image decoding device (100) cannot divide the square-shaped first encoding unit (1100) into four square-shaped second encoding units (1130a, 1130b, 1130c, 1130d). Based on the segmented form mode information, the image decoding device (100) can determine a non-square second encoding unit (1110a, 1110b, 1120a, 1120b, etc.).

[0160] According to one embodiment of the present disclosure, the image decoding device (100) can independently divide each of the second encoding units (1110a, 1110b, 1120a, 1120b, etc.) in a non-square shape. Each of the second encoding units (1110a, 1110b, 1120a, 1120b, etc.) can be divided in a predetermined order through a recursive method, and this may be a division method corresponding to the method in which the first encoding unit (1100) is divided based on division shape mode information.

[0161] For example, the image decoding device (100) can determine a square-shaped third encoding unit (1112a, 1112b) by dividing the left second encoding unit (1110a) in a horizontal direction, and can determine a square-shaped third encoding unit (1114a, 1114b) by dividing the right second encoding unit (1110b) in a horizontal direction. Furthermore, the image decoding device (100) can determine a square-shaped third encoding unit (1116a, 1116b, 1116c, 1116d) by dividing both the left second encoding unit (1110a) and the right second encoding unit (1110b) in a horizontal direction. In this case, the encoding unit can be determined in the same form as the first encoding unit (1100) being divided into four square-shaped second encoding units (1130a, 1130b, 1130c, 1130d).

[0162] As another example, the video decoding device (100) may determine a square-shaped third encoding unit (1122a, 1122b) by dividing the upper second encoding unit (1120a) in a vertical direction, and determine a square-shaped third encoding unit (1124a, 1124b) by dividing the lower second encoding unit (1120b) in a vertical direction. Furthermore, the video decoding device (100) may determine a square-shaped third encoding unit (1126a, 1126b, 1126a, 1126b) by dividing both the upper second encoding unit (1120a) and the lower second encoding unit (1120b) in a vertical direction. In this case, the encoding unit can be determined in the same form as the first encoding unit (1100) being divided into four square-shaped second encoding units (1130a, 1130b, 1130c, 1130d).

[0163] FIG. 12 illustrates that, according to one embodiment of the present disclosure, the processing order between a plurality of encoding units may vary depending on the division process of the encoding unit.

[0164] According to one embodiment of the present disclosure, an image decoding device (100) may divide a first encoding unit (1200) based on division shape mode information. When the block shape is square and the division shape mode information indicates that the first encoding unit (1200) is divided in at least one of a horizontal direction and a vertical direction, the image decoding device (100) may divide the first encoding unit (1200) to determine a second encoding unit (e.g., 1210a, 1210b, 1220a, 1220b, etc.). Referring to FIG. 12, the second encoding units (1210a, 1210b, 1220a, 1220b) in a non-square shape determined by dividing the first encoding unit (1200) only in a horizontal direction or a vertical direction may be divided independently based on the division shape mode information for each. For example, the video decoding device (100) can determine the third encoding unit (1216a, 1216b, 1216c, 1216d) by dividing the second encoding unit (1210a, 1210b) generated by dividing the first encoding unit (1200) in the vertical direction, and can determine the third encoding unit (1226a, 1226b, 1226c, 1226d) by dividing the second encoding unit (1220a, 1220b) generated by dividing the first encoding unit (1200) in the horizontal direction, respectively, in the vertical direction. Since the process of dividing these second encoding units (1210a, 1210b, 1220a, 1220b) has been described in detail in relation to FIG. 11, a detailed explanation will be omitted.

[0165] According to one embodiment of the present disclosure, the image decoding device (100) can process encoding units in a predetermined order. Since the characteristics of processing encoding units in a predetermined order have been described in detail in relation to FIG. 7, a detailed description will be omitted. Referring to FIG. 12, the image decoding device (100) can divide a square-shaped first encoding unit (1200) to determine four square-shaped third encoding units (1216a, 1216b, 1216c, 1216d, 1226a, 1226b, 1226c, 1226d). According to one embodiment of the present disclosure, the image decoding device (100) can determine the processing order of the third encoding unit (1216a, 1216b, 1216c, 1216d, 1226a, 1226b, 1226c, 1226d) according to the form in which the first encoding unit (1200) is divided.

[0166] According to one embodiment of the present disclosure, the image decoding device (100) can determine the third encoding unit (1216a, 1216b, 1216c, 1216d) by dividing the second encoding unit (1210a, 1210b) generated by dividing in a vertical direction into the horizontal direction, and the image decoding device (100) can process the third encoding unit (1216a, 1216c) included in the left second encoding unit (1210a) first in a vertical direction, and then process the third encoding unit (1216b, 1216d) included in the right second encoding unit (1210b) in a vertical direction according to the order (1217).

[0167] According to one embodiment of the present disclosure, the image decoding device (100) can determine the third encoding unit (1226a, 1226b, 1226c, 1226d) by dividing the second encoding unit (1220a, 1220b) generated by dividing in a horizontal direction into the vertical direction, and the image decoding device (100) can process the third encoding unit (1226a, 1226b, 1226c, 1226d) according to the order (1227) of first processing the third encoding unit (1226a, 1226b) included in the upper second encoding unit (1220a) in a horizontal direction, and then processing the third encoding unit (1226c, 1226d) included in the lower second encoding unit (1220b) in a horizontal direction.

[0168] Referring to FIG. 12, the second encoding units (1210a, 1210b, 1220a, 1220b) are each divided to determine the square-shaped third encoding units (1216a, 1216b, 1216c, 1216d, 1226a, 1226b, 1226c, 1226d). The second encoding unit (1210a, 1210b) determined by dividing in the vertical direction and the second encoding unit (1220a, 1220b) determined by dividing in the horizontal direction are divided into different forms, but according to the third encoding unit (1216a, 1216b, 1216c, 1216d, 1226a, 1226b, 1226c, 1226d) determined thereafter, the result is that the first encoding unit (1200) is divided into encoding units of the same form. Accordingly, the image decoding device (100) recursively divides the encoding unit through different processes based on the division form mode information, so that even if the encoding units of the same form are determined as a result, the multiple encoding units determined in the same form can be processed in different orders.

[0169] FIG. 13 illustrates a process in which, according to one embodiment of the present disclosure, a encoding unit is recursively divided to determine a plurality of encoding units, and the depth of the encoding unit is determined as the shape and size of the encoding unit change.

[0170] According to one embodiment of the present disclosure, the image decoding device (100) may determine the depth of an encoding unit according to a predetermined standard. For example, the predetermined standard may be the length of the long side of the encoding unit. The image decoding device (100) may determine that if the length of the long side of the current encoding unit is divided by 2n (n>0) times the length of the long side of the encoding unit before division, the depth of the current encoding unit is increased by n compared to the depth of the encoding unit before division. In the following, the encoding unit with increased depth is expressed as an encoding unit of lower depth.

[0171] Referring to FIG. 13, according to one embodiment of the present disclosure, based on block shape information indicating that it is a square shape (for example, the block shape information may indicate '0: SQUARE'), an image decoding device (100) can determine a second encoding unit (1302), a third encoding unit (1304), etc. of a lower depth by dividing a first encoding unit (1300) that is square in shape. If the size of the first encoding unit (1300) that is square in shape is 2Nx2N, the second encoding unit (1302), which is determined by dividing the width and height of the first encoding unit (1300) by half, may have a size of NxN. Furthermore, the third encoding unit (1304), which is determined by dividing the width and height of the second encoding unit (1302) by half, may have a size of N / 2xN / 2. In this case, the width and height of the third encoding unit (1304) correspond to 1 / 4 times the width and height of the first encoding unit (1300). When the depth of the first encoding unit (1300) is D, the depth of the second encoding unit (1302), which is 1 / 2 times the width and height of the first encoding unit (1300), may be D+1, and the depth of the third encoding unit (1304), which is 1 / 4 times the width and height of the first encoding unit (1300), may be D+2.

[0172] According to one embodiment of the present disclosure, based on block shape information representing a non-square shape (for example, the block shape information may represent '1: NS_VER' indicating that the height is longer than the width, or '2: NS_HOR' indicating that the width is longer than the height), an image decoding device (100) may divide a first encoding unit (1310 or 1320) that is a non-square shape to determine a second encoding unit (1312 or 1322), a third encoding unit (1314 or 1324), etc. of a lower depth.

[0173] The image decoding device (100) can determine a second encoding unit (e.g., 1302, 1312, 1322, etc.) by dividing at least one of the width and height of a first encoding unit (1310) of size Nx2N. That is, the image decoding device (100) can determine a second encoding unit (1302) of size NxN or a second encoding unit (1322) of size NxN / 2 by dividing the first encoding unit (1310) in a horizontal direction, and can also determine a second encoding unit (1312) of size N / 2xN by dividing it in a horizontal direction and a vertical direction.

[0174] According to one embodiment of the present disclosure, the image decoding device (100) may determine a second encoding unit (e.g., 1302, 1312, 1322, etc.) by dividing at least one of the width and height of a first encoding unit (1320) of size 2NxN. That is, the image decoding device (100) may determine a second encoding unit (1302) of size NxN or a second encoding unit (1312) of size N / 2xN by dividing the first encoding unit (1320) in the vertical direction, and may determine a second encoding unit (1322) of size NxN / 2 by dividing it in the horizontal and vertical directions.

[0175] According to one embodiment of the present disclosure, the image decoding device (100) may determine a third encoding unit (e.g., 1304, 1314, 1324, etc.) by dividing at least one of the width and height of a second encoding unit (1302) of size NxN. That is, the image decoding device (100) may determine a third encoding unit (1304) of size N / 2xN / 2 by dividing the second encoding unit (1302) in a vertical direction and a horizontal direction, or determine a third encoding unit (1314) of size N / 4xN / 2, or determine a third encoding unit (1324) of size N / 2xN / 4.

[0176] According to one embodiment of the present disclosure, the image decoding device (100) may determine a third encoding unit (e.g., 1304, 1314, 1324, etc.) by dividing at least one of the width and height of a second encoding unit (1312) of size N / 2xN. That is, the image decoding device (100) may determine a third encoding unit (1304) of size N / 2xN / 2 or a third encoding unit (1324) of size N / 2xN / 4 by dividing the second encoding unit (1312) in a horizontal direction, or determine a third encoding unit (1314) of size N / 4xN / 2 by dividing it in a vertical direction and a horizontal direction.

[0177] According to one embodiment of the present disclosure, the image decoding device (100) may determine a third encoding unit (e.g., 1304, 1314, 1324, etc.) by dividing at least one of the width and height of a second encoding unit (1322) of size NxN / 2. That is, the image decoding device (100) may determine a third encoding unit (1304) of size N / 2xN / 2 or a third encoding unit (1314) of size N / 4xN / 2 by dividing the second encoding unit (1322) in the vertical direction, or determine a third encoding unit (1324) of size N / 2xN / 4 by dividing it in the vertical and horizontal directions.

[0178] According to one embodiment of the present disclosure, an image decoding device (100) may divide square-shaped encoding units (e.g., 1300, 1302, 1304) in a horizontal or vertical direction. For example, a first encoding unit (1300) of size 2Nx2N may be divided in a vertical direction to determine a first encoding unit (1310) of size Nx2N, or divided in a horizontal direction to determine a first encoding unit (1320) of size 2NxN. According to one embodiment of the present disclosure, when the depth is determined based on the length of the longest side of the encoding unit, the depth of the encoding unit determined by dividing the first encoding unit (1300) of size 2Nx2N in a horizontal or vertical direction may be the same as the depth of the first encoding unit (1300).

[0179] According to one embodiment of the present disclosure, the width and height of the third encoding unit (1314 or 1324) may correspond to 1 / 4 times the width and height of the first encoding unit (1310 or 1320). If the depth of the first encoding unit (1310 or 1320) is D, the depth of the second encoding unit (1312 or 1322), which is 1 / 2 times the width and height of the first encoding unit (1310 or 1320), may be D+1, and the depth of the third encoding unit (1314 or 1324), which is 1 / 4 times the width and height of the first encoding unit (1310 or 1320), may be D+2.

[0180] FIG. 14 illustrates a depth and part index (hereinafter PID) for distinguishing between coding units that can be determined according to the shape and size of the coding units according to one embodiment of the present disclosure.

[0181] According to one embodiment of the present disclosure, an image decoding device (100) can determine various shapes of second encoding units by dividing a square-shaped first encoding unit (1400). Referring to FIG. 14, the image decoding device (100) can determine second encoding units (1402a, 1402b, 1404a, 1404b, 1406a, 1406b, 1406c, 1406d) by dividing the first encoding unit (1400) in at least one of a vertical direction and a horizontal direction according to the division shape mode information. That is, the image decoding device (100) can determine the second encoding unit (1402a, 1402b, 1404a, 1404b, 1406a, 1406b, 1406c, 1406d) based on the segmented form mode information for the first encoding unit (1400).

[0182] According to one embodiment of the present disclosure, the depth of the second encoding unit (1402a, 1402b, 1404a, 1404b, 1406a, 1406b, 1406c, 1406d), which is determined according to the segmented shape mode information for the first encoding unit (1400) in a square shape, can be determined based on the length of the longer side. For example, since the length of one side of the first encoding unit (1400) in a square shape and the length of the longer side of the second encoding unit (1402a, 1402b, 1404a, 1404b) in a non-square shape are the same, the depth of the first encoding unit (1400) and the second encoding unit (1402a, 1402b, 1404a, 1404b) in a non-square shape can be considered to be the same as D. In contrast, when the video decoding device (100) divides the first encoding unit (1400) into four square-shaped second encoding units (1406a, 1406b, 1406c, 1406d) based on the division shape mode information, since the length of one side of the square-shaped second encoding units (1406a, 1406b, 1406c, 1406d) is half the length of one side of the first encoding unit (1400), the depth of the second encoding units (1406a, 1406b, 1406c, 1406d) may be a depth of D+1, which is one depth lower than the depth D of the first encoding unit (1400).

[0183] According to one embodiment of the present disclosure, an image decoding device (100) may divide a first encoding unit (1410), in which the height is longer than the width, into a plurality of second encoding units (1412a, 1412b, 1414a, 1414b, 1414c) by dividing it in a horizontal direction according to the division shape mode information. According to one embodiment of the present disclosure, an image decoding device (100) may divide a first encoding unit (1420), in which the width is longer than the height, into a plurality of second encoding units (1422a, 1422b, 1424a, 1424b, 1424c) by dividing it in a vertical direction according to the division shape mode information.

[0184] According to one embodiment of the present disclosure, a second encoding unit (1412a, 1412b, 1414a, 1414b, 1414c, 1422a, 1422b, 1424a, 1424b, 1424c) determined according to the segmented shape mode information for a first encoding unit (1410 or 1420) of a non-square shape may have its depth determined based on the length of the longer side. For example, since the length of one side of the square-shaped second encoding unit (1412a, 1412b) is half the length of one side of the non-square-shaped first encoding unit (1410) in which the height is longer than the width, the depth of the square-shaped second encoding unit (1412a, 1412b) is D+1, which is one depth lower than the depth D of the non-square-shaped first encoding unit (1410).

[0185] Furthermore, the image decoding device (100) may divide a first encoding unit (1410) in a non-square shape into an odd number of second encoding units (1414a, 1414b, 1414c) based on the division shape mode information. The odd number of second encoding units (1414a, 1414b, 1414c) may include a second encoding unit (1414a, 1414c) in a non-square shape and a second encoding unit (1414b) in a square shape. In this case, since the length of the longer side of the non-square second encoding unit (1414a, 1414c) and the length of one side of the square second encoding unit (1414b) are half the length of one side of the first encoding unit (1410), the depth of the second encoding unit (1414a, 1414b, 1414c) may be a depth of D+1, which is one depth lower than the depth D of the first encoding unit (1410). The image decoding device (100) may determine the depth of the encoding units associated with the non-square first encoding unit (1420), in a manner corresponding to the above method of determining the depth of the encoding units associated with the first encoding unit (1410).

[0186] According to one embodiment of the present disclosure, an image decoding device (100) may determine an index (PID) for distinguishing divided encoding units, and when the odd number of divided encoding units are not of the same size, the index may be determined based on the size ratio between the encoding units. Referring to FIG. 14, among the odd number of divided encoding units (1414a, 1414b, 1414c), the encoding unit (1414b) located in the middle may have the same width as the other encoding units (1414a, 1414c) but may have a height twice that of the other encoding units (1414a, 1414c). That is, in this case, the encoding unit (1414b) located in the middle may include two of the other encoding units (1414a, 1414c). Accordingly, if the index (PID) of the encoding unit (1414b) located in the middle according to the scan order is 1, the encoding unit (1414c) located in the next order may have an index of 3, which is an increase of 2. That is, there may be a discontinuity in the value of the index. According to one embodiment of the present disclosure, the image decoding device (100) may determine whether the encoding units divided into an odd number are not of the same size based on whether there is a discontinuity in the index for distinguishing between these divided encoding units.

[0187] According to one embodiment of the present disclosure, an image decoding device (100) may determine whether a plurality of encoding units determined by dividing from a current encoding unit are divided into a specific division form based on the value of an index for distinguishing the plurality of encoding units. Referring to FIG. 14, the image decoding device (100) may divide a first encoding unit (1410) in the shape of a rectangle whose height is greater than its width to determine an even number of encoding units (1412a, 1412b) or an odd number of encoding units (1414a, 1414b, 1414c). The image decoding device (100) may use an index (PID) representing each encoding unit to distinguish each of the plurality of encoding units. According to one embodiment of the present disclosure, the PID may be obtained from a sample at a predetermined position of each encoding unit (e.g., the upper left sample).

[0188] According to one embodiment of the present disclosure, an image decoding device (100) may determine a coding unit at a predetermined position among the coding units determined by division using an index for distinguishing coding units. According to one embodiment of the present disclosure, if the division shape mode information for a first coding unit (1410) in the form of a rectangle whose height is longer than its width indicates that it is divided into three coding units, the image decoding device (100) may divide the first coding unit (1410) into three coding units (1414a, 1414b, 1414c). The image decoding device (100) may assign an index to each of the three coding units (1414a, 1414b, 1414c). The image decoding device (100) may compare the indices for each coding unit to determine the middle coding unit among the odd number of divided coding units. The image decoding device (100) may determine a encoding unit (1414b) having an index corresponding to the middle value among the indices based on the indices of the encoding units, as the encoding unit at the middle position among the encoding units determined by dividing the first encoding unit (1410). According to one embodiment of the present disclosure, when determining an index for distinguishing the divided encoding units, the image decoding device (100) may determine the index based on the size ratio between the encoding units if the encoding units are not of the same size. Referring to FIG. 14, the encoding unit (1414b) generated by dividing the first encoding unit (1410) may have the same width as the other encoding units (1414a, 1414c) but may be twice the height of the encoding units (1414a, 1414c) which have different heights. In this case, if the index (PID) of the encoding unit (1414b) located in the middle is 1, the encoding unit (1414c) located in the next order may have an index of 3, which is increased by 2.In cases where the index increases uniformly and then the rate of increase changes, such as in this case, the image decoding device (100) may determine that the current encoding unit is divided into a plurality of encoding units, including encoding units having different sizes from other encoding units. In the case where the division type mode information indicates that the current encoding unit is divided into an odd number of encoding units according to one embodiment of the present disclosure, the image decoding device (100) may divide the current encoding unit in such a way that the encoding unit at a predetermined position among the odd number of encoding units (e.g., the middle encoding unit) has a different size from other encoding units. In this case, the image decoding device (100) may determine the middle encoding unit having a different size by using an index (PID) for the encoding unit. However, the aforementioned index, the size or position of the encoding unit at a predetermined position to be determined, is specific for the purpose of explaining one embodiment and should not be interpreted as being limited thereto, and should be interpreted as allowing various indices, positions, and sizes of encoding units to be used.

[0189] According to one embodiment of the present disclosure, an image decoding device (100) may use a predetermined data unit in which recursive division of the encoding unit begins.

[0190] FIG. 15 illustrates that a plurality of encoding units are determined according to a plurality of predetermined data units included in a picture according to one embodiment of the present disclosure.

[0191] According to one embodiment of the present disclosure, a predetermined data unit may be defined as a data unit in which an encoding unit begins to recursively divide using partitioning mode information. That is, it may correspond to the highest depth encoding unit used in the process of determining a plurality of encoding units that divide the current picture. For convenience of explanation, such a predetermined data unit will be referred to as a reference data unit below.

[0192] According to one embodiment of the present disclosure, a reference data unit may have a predetermined size and shape. According to one embodiment of the present disclosure, the reference data unit may include MxN samples. Here, M and N may be the same and may be integers expressed as powers of 2. That is, the reference data unit may have a square or non-square shape and may subsequently be divided into an integer number of encoding units.

[0193] According to one embodiment of the present disclosure, the image decoding device (100) can divide the current picture into a plurality of reference data units. According to one embodiment of the present disclosure, the image decoding device (100) can divide the current picture into a plurality of reference data units using division form mode information for each reference data unit. This division process of reference data units may correspond to a division process using a quad-tree structure.

[0194] According to one embodiment of the present disclosure, the image decoding device (100) can predetermine the minimum size that a reference data unit included in the current picture may have. Accordingly, the image decoding device (100) can determine reference data units of various sizes having a size greater than or equal to the minimum size, and can determine at least one encoding unit using segmented form mode information based on the determined reference data unit.

[0195] Referring to FIG. 15, the image decoding device (100) may use a square-shaped reference encoding unit (1500) or a non-square-shaped reference encoding unit (1502). According to one embodiment of the present disclosure, the shape and size of the reference encoding unit may be determined according to various data units (e.g., sequence, picture, slice, slice segment, tile, tile group, maximum encoding unit, etc.) that may include at least one reference encoding unit.

[0196] According to one embodiment of the present disclosure, the bitstream acquisition unit (110) of the image decoding device (100) can acquire at least one of information regarding the shape of the reference encoding unit and information regarding the size of the reference encoding unit from the bitstream for each of the various data units. The process of determining at least one encoding unit included in the square-shaped reference encoding unit (1500) has been described in detail through the process of dividing the current encoding unit (300) of FIG. 3, and the process of determining at least one encoding unit included in the non-square-shaped reference encoding unit (1502) has been described in detail through the process of dividing the current encoding unit (400 or 450) of FIG. 4, so a detailed explanation is omitted.

[0197] According to one embodiment of the present disclosure, an image decoding device (100) may use an index to identify the size and shape of a reference coding unit in order to determine the size and shape of a reference coding unit according to a portion of data units that are predetermined based on a predetermined condition. That is, a bitstream acquisition unit (110) may acquire only an index for identifying the size and shape of a reference coding unit for each slice, slice segment, tile, tile group, maximum coding unit, etc., among the various data units (e.g., sequence, picture, slice, slice segment, tile, tile group, maximum coding unit, etc.) from the bitstream, as a data unit that satisfies a predetermined condition (e.g., a data unit having a size smaller than or equal to a slice). By using the index, the image decoding device (100) may determine the size and shape of a reference data unit for each data unit that satisfies the predetermined condition. When information regarding the form of the reference encoding unit and information regarding the size of the reference encoding unit are obtained from the bitstream for each data unit of a relatively small size and used, the utilization efficiency of the bitstream may be poor; therefore, instead of directly obtaining information regarding the form of the reference encoding unit and information regarding the size of the reference encoding unit, only the index may be obtained and used. In this case, at least one of the size and form of the reference encoding unit corresponding to the index representing the size and form of the reference encoding unit may be predetermined. That is, the image decoding device (100) can determine at least one of the size and form of the reference encoding unit included in the data unit that serves as the basis for obtaining the index by selecting at least one of the predetermined size and form of the reference encoding unit according to the index.

[0198] According to one embodiment of the present disclosure, the image decoding device (100) may utilize at least one reference encoding unit included in one maximum encoding unit (1510). That is, the maximum encoding unit (1510) that divides the image may include at least one reference encoding unit, and the encoding unit may be determined through a recursive division process of each reference encoding unit. According to one embodiment of the present disclosure, at least one of the width and height of the maximum encoding unit (1510) may correspond to an integer multiple of at least one of the width and height of the reference encoding unit. According to one embodiment of the present disclosure, the size of the reference encoding unit may be the size obtained by dividing the maximum encoding unit (1510) n times according to a quad tree structure. That is, the image decoding device (100) may determine the reference encoding unit by dividing the maximum encoding unit (1510) n times according to a quad tree structure, and according to one embodiment of the present disclosure, the reference encoding unit may be divided based on at least one of block shape information and division shape mode information.

[0199] According to one embodiment of the present disclosure, an image decoding device (100) may obtain and use block form information indicating the form of a current encoding unit or division form mode information indicating a method of dividing a current encoding unit from a bitstream. The division form mode information may be included in a bitstream associated with various data units. For example, the image decoding device (100) may use the division form mode information included in a sequence parameter set, a picture parameter set, a video parameter set, a slice header, a slice segment header, a tile header, and a tile group header. Furthermore, the image decoding device (100) may obtain and use syntax elements corresponding to the block form information or division form mode information from the bitstream for each maximum encoding unit and reference encoding unit.

[0200] A method for determining a division rule according to one embodiment of the present disclosure will be described in detail below.

[0201] The image decoding device (100) can determine the segmentation rule of the image. The segmentation rule may be predetermined between the image decoding device (100) and the image encoding device (200). The image decoding device (100) can determine the segmentation rule of the image based on information obtained from a bitstream. The image decoding device (100) can determine the segmentation rule based on information obtained from at least one of a sequence parameter set, a picture parameter set, a video parameter set, a slice header, a slice segment header, a tile header, and a tile group header. The image decoding device (100) can determine the segmentation rule differently depending on the frame, slice, tile, temporal layer, maximum encoding unit, or encoding unit.

[0202] The image decoding device (100) can determine a partitioning rule based on the block shape of the encoding unit. The block shape may include the size, shape, ratio of width and height, and orientation of the encoding unit. The image encoding device (200) and the image decoding device (100) may predetermine to determine a partitioning rule based on the block shape of the encoding unit. However, they are not limited thereto. The image decoding device (100) can determine a partitioning rule based on information obtained from a bitstream received from the image encoding device (200).

[0203] The shape of the encoding unit may include square and non-square. If the width and height of the encoding unit are the same, the image decoder (100) may determine the shape of the encoding unit as square. Additionally, if the width and height of the encoding unit are not the same, the image decoder (100) may determine the shape of the encoding unit as non-square.

[0204] The size of the encoding unit may include various sizes such as 4x4, 8x4, 4x8, 8x8, 16x4, 16x8, ..., 256x256. The size of the encoding unit may be classified according to the length of the long side, the length of the short side, or the width of the encoding unit. The image decoding device (100) may apply the same partitioning rule to encoding units classified into the same group. For example, the image decoding device (100) may classify encoding units having the same long side length into the same size. Additionally, the image decoding device (100) may apply the same partitioning rule to encoding units having the same long side length.

[0205] The ratio of width to height of a encoding unit may include 1:2, 2:1, 1:4, 4:1, 1:8, 8:1, 1:16, 16:1, 32:1, or 1:32, etc. Additionally, the direction of the encoding unit may include a horizontal direction and a vertical direction. The horizontal direction may indicate a case where the width of the encoding unit is longer than the height. The vertical direction may indicate a case where the width of the encoding unit is shorter than the height.

[0206] The image decoder (100) can adaptively determine a splitting rule based on the size of the encoding unit. The image decoder (100) can determine different acceptable splitting mode types based on the size of the encoding unit. For example, the image decoder (100) can determine whether splitting is allowed based on the size of the encoding unit. The image decoder (100) can determine the splitting direction according to the size of the encoding unit. The image decoder (100) can determine an acceptable splitting type according to the size of the encoding unit.

[0207] Determining the division rule based on the size of the encoding unit may be a division rule predetermined between the image encoding device (200) and the image decoding device (100). Additionally, the image decoding device (100) may determine the division rule based on information obtained from the bitstream.

[0208] The image decoder (100) can adaptively determine a partitioning rule based on the position of the encoding unit. The image decoder (100) can adaptively determine a partitioning rule based on the position occupied by the encoding unit in the image.

[0209] Additionally, the video decoding device (100) can determine a splitting rule so that encoding units generated by different splitting paths do not have the same block shape. However, this is not limited thereto, and encoding units generated by different splitting paths may have the same block shape. Encoding units generated by different splitting paths may have different decoding processing orders. Since the decoding processing order has been explained together with FIG. 12, a detailed explanation is omitted.

[0210] FIG. 16 illustrates encoding units that can be determined for each picture when the combination of forms in which encoding units can be divided according to one embodiment of the present disclosure is different for each picture.

[0211] Referring to FIG. 16, the image decoding device (100) may determine different combinations of division forms in which the encoding unit can be divided for each picture. For example, the image decoding device (100) may decode an image using a picture (1600) that can be divided into four encoding units, a picture (1610) that can be divided into two or four encoding units, and a picture (1620) that can be divided into two, three, or four encoding units among at least one picture included in the image. To divide the picture (1600) into multiple encoding units, the image decoding device (100) may use only division form information indicating that it is divided into four square encoding units. To divide the picture (1610), the image decoding device (100) may use only division form information indicating that it is divided into two or four encoding units. The image decoding device (100) may use only division form information indicating that the picture (1620) is divided into two, three, or four encoding units in order to divide the picture. Since the combination of division forms described above is merely an example for explaining the operation of the image decoding device (100), the combination of division forms described above should not be interpreted as being limited to the above example, but should be interpreted as allowing various combinations of division forms to be used for each predetermined data unit.

[0212] According to one embodiment of the present disclosure, a bitstream acquisition unit (110) of an image decoding device (100) may acquire a bitstream including an index representing a combination of division form information for each predetermined data unit (e.g., sequence, picture, slice, slice segment, tile, or tile group, etc.). For example, the bitstream acquisition unit (110) may acquire an index representing a combination of division form information from a sequence parameter set, a picture parameter set, a slice header, a tile header, or a tile group header. The image decoding device (100) may determine a combination of division forms in which the encoding unit can be divided for each predetermined data unit using the acquired index, and accordingly, different combinations of division forms may be used for each predetermined data unit.

[0213] FIG. 17 illustrates various forms of encoding units that can be determined based on partitioned form mode information that can be expressed as binary code according to one embodiment of the present disclosure.

[0214] According to one embodiment of the present disclosure, the image decoding device (100) can divide the encoding unit into various forms using block form information and division form mode information obtained through the bitstream acquisition unit (110). The forms of the encoding unit that can be divided may correspond to various forms including the forms described through the embodiments described above.

[0215] Referring to FIG. 17, the image decoding device (100) can divide a square-shaped encoding unit into at least one of a horizontal direction and a vertical direction based on the divided shape mode information, and can divide a non-square-shaped encoding unit into a horizontal direction or a vertical direction.

[0216] According to one embodiment of the present disclosure, if an image decoding device (100) can divide a square-shaped encoding unit into four square encoding units by dividing it in the horizontal and vertical directions, there may be four types of division forms that can be represented by the division form mode information for the square encoding unit. According to one embodiment of the present disclosure, the division form mode information may be expressed as a two-digit binary code, and a binary code may be assigned to each division form. For example, if the encoding unit is not divided, the division form mode information may be expressed as (00)b; if the encoding unit is divided in the horizontal and vertical directions, the division form mode information may be expressed as (01)b; if the encoding unit is divided in the horizontal direction, the division form mode information may be expressed as (10)b; and if the encoding unit is divided in the vertical direction, the division form mode information may be expressed as (11)b.

[0217] According to one embodiment of the present disclosure, when the image decoding device (100) divides a non-square-shaped encoding unit in a horizontal or vertical direction, the type of division form that can be represented by the division form mode information may be determined by how many encoding units are divided. Referring to FIG. 17, the image decoding device (100) may divide a non-square-shaped encoding unit up to three times according to one embodiment of the present disclosure. The image decoding device (100) may divide the encoding unit into two encoding units, in which case the division form mode information may be expressed as (10)b. The image decoding device (100) may divide the encoding unit into three encoding units, in which case the division form mode information may be expressed as (11)b. The image decoding device (100) may decide not to divide the encoding unit, in which case the division form mode information may be expressed as (0)b. That is, the image decoding device (100) can use variable length coding (VLC) rather than fixed length coding (FLC) to use binary code representing segmented form mode information.

[0218] Referring to FIG. 17 according to one embodiment of the present disclosure, the binary code of the division form mode information indicating that the encoding unit is not divided can be represented as (0)b. If the binary code of the division form mode information indicating that the encoding unit is not divided is set to (00)b, then all 2 bits of the binary code of the division form mode information must be used even though there is no division form mode information set to (01)b. However, as illustrated in FIG. 17, if three division forms for a non-square encoding unit are used, the image decoding device (100) can determine that the encoding unit is not divided even if a 1-bit binary code (0)b is used as the division form mode information, thus allowing for efficient use of the bitstream. However, the division form of the non-square encoding unit indicated by the division form mode information should not be interpreted as being limited only to the three forms illustrated in FIG. 17, but should be interpreted as various forms including the embodiments described above.

[0219] FIG. 18 illustrates another form of encoding unit that can be determined based on partitioned form mode information that can be represented as a binary code according to one embodiment of the present disclosure.

[0220] Referring to FIG. 18, the image decoding device (100) can divide a square-shaped encoding unit in a horizontal or vertical direction based on the division shape mode information, and can divide a non-square-shaped encoding unit in a horizontal or vertical direction. That is, the division shape mode information can indicate that a square-shaped encoding unit is divided in one direction. In this case, the binary code of the division shape mode information indicating that the square-shaped encoding unit is not divided can be represented as (0)b. If the binary code of the division shape mode information indicating that the encoding unit is not divided is set to (00)b, then all 2 bits of the binary code of the division shape mode information must be used even though there is no division shape mode information set to (01)b. However, as illustrated in FIG. 18, if three division forms for a square-shaped encoding unit are used, the image decoding device (100) can determine that the encoding unit is not divided even if a 1-bit binary code (0)b is used as division form mode information, thus allowing the bitstream to be used efficiently. However, the division form of the square-shaped encoding unit indicated by the division form mode information should not be interpreted as being limited only to the three forms illustrated in FIG. 18, but should be interpreted as various forms including the embodiments described above.

[0221] According to one embodiment of the present disclosure, block form information or partition form mode information can be represented using binary code, and such information can be immediately generated as a bitstream. Additionally, block form information or partition form mode information that can be represented by binary code may not be immediately generated as a bitstream but may be used as binary code input in CABAC (context adaptive binary arithmetic coding).

[0222] According to one embodiment of the present disclosure, an image decoding device (100) describes a process of obtaining syntax for block form information or partition form mode information through CABAC. A bitstream including a binary code for the syntax can be obtained through a bitstream acquisition unit (110). The image decoding device (100) can detect a syntax element representing block form information or partition form mode information by debinding a bin string included in the obtained bitstream. According to one embodiment of the present disclosure, the image decoding device (100) obtains a set of binary bin strings corresponding to the syntax element to be decoded, and can decode each bin using probability information, and the image decoding device (100) can repeat this process until the bin string composed of these decoded bins becomes equal to one of the previously obtained bin strings. The image decoding device (100) can determine syntax elements by performing inverse binarization of an empty string.

[0223] According to one embodiment of the present disclosure, an image decoding device (100) can determine the syntax for a bin string by performing a decoding process of adaptive binary arithmetic coding, and the image decoding device (100) can update a probability model for bins obtained through a bitstream acquisition unit (110). Referring to FIG. 17, the bitstream acquisition unit (110) of the image decoding device (100) can acquire a bitstream representing a binary code representing partitioned mode information according to one embodiment of the present disclosure. Using the acquired binary code having a size of 1 bit or 2 bits, the image decoding device (100) can determine the syntax for the partitioned mode information. To determine the syntax for the partitioned mode information, the image decoding device (100) can update the probability for each bit of the 2-bit binary code. That is, the image decoding device (100) can update the probability of having a value of 0 or 1 when decoding the next bin, depending on whether the value of the first bin of the 2-bit binary code is 0 or 1.

[0224] According to one embodiment of the present disclosure, the image decoding device (100) can update the probability for the bins used in the process of decoding the bins of the empty string for the syntax during the process of determining the syntax, and the image decoding device (100) can determine that the probability is not updated for certain bits of the empty string and has the same probability.

[0225] Referring to FIG. 17, in the process of determining syntax using an empty string representing segmentation mode information for a non-square type encoding unit, the image decoder (100) can determine syntax for the segmentation mode information using one bin having a value of 0 when the non-square type encoding unit is not segmented. That is, when block type information indicates that the current encoding unit is a non-square type, the first bin of the empty string for the segmentation mode information may be 0 when the non-square type encoding unit is not segmented, and 1 when it is segmented into two or three encoding units. Accordingly, the probability that the first bin of the empty string for the segmentation mode information for a non-square type encoding unit is 0 may be 1 / 3, and the probability that it is 1 may be 2 / 3. As described above, since the image decoding device (100) can represent only a 1-bit empty string having a value of 0 as the segmentation form mode information indicating that a non-square type encoding unit is not segmented, the image decoding device (100) can determine the syntax for the segmentation form mode information by determining whether the second bin is 0 or 1 only when the first bin of the segmentation form mode information is 1. According to one embodiment of the present disclosure, the image decoding device (100) can decode the bin by considering that when the first bin of the segmentation form mode information is 1, the probability that the second bin is 0 or 1 is equal to the probability.

[0226] According to one embodiment of the present disclosure, the image decoding device (100) may use various probabilities for each bin in the process of determining the bins of the bin string for the segmented form mode information. According to one embodiment of the present disclosure, the image decoding device (100) may determine the probability of the bins for the segmented form mode information differently depending on the direction of the non-square block. According to one embodiment of the present disclosure, the image decoding device (100) may determine the probability of the bins for the segmented form mode information differently depending on the width or the length of the long side of the current encoding unit. According to one embodiment of the present disclosure, the image decoding device (100) may determine the probability of the bins for the segmented form mode information differently depending on at least one of the shape of the current encoding unit and the length of the long side.

[0227] According to one embodiment of the present disclosure, the image decoding device (100) may determine that the probability of bins for segmented form mode information is the same for encoding units of a predetermined size or larger. For example, based on the length of the long side of the encoding unit, the probability of bins for segmented form mode information is determined to be the same for encoding units of a size of 64 samples or larger.

[0228] According to one embodiment of the present disclosure, the image decoding device (100) may determine the initial probability for the bins constituting the empty string of the segmented form mode information based on the slice type (e.g., I slice, P slice, or B slice).

[0229] Figure 19 is a block diagram of an image encoding and decoding system.

[0230] The encoding unit (1910) of the video encoding and decoding system (1900) transmits an encoded bitstream of the video, and the decoding unit (1950) receives the bitstream and decodes it to output a restored video. Here, the encoding unit (1910) may have a configuration similar to the video encoding device (200) described later, and the decoding unit (1950) may have a configuration similar to the video decoding device (100).

[0231] In the encoding unit (1910), the prediction encoding unit (1915) outputs prediction data through inter-prediction and intra-prediction, and the transformation and quantization unit (1920) outputs quantized transformed coefficients (or quantized coefficients) of residual data between the prediction data and the current input image. For example, the transform coefficients may be generated using a transformation kernel that includes at least one of the Discrete Cosine Transform (DT), Discrete Sine Transform (DST), Hadamard transform, Karhunen-Loeve Transform (KLT), or Wavelet Transform. The residual data may have its information compressed as the transformation is performed. For example, the residual data may be represented using a small number of frequencies as the transformation is performed. In one embodiment, the conversion and quantization unit (1920) may omit the conversion of residual data between the prediction data and the current input image. For example, if the distribution of residual data makes it inefficient to perform the conversion, the conversion process for residual data may be omitted and the quantization process may be performed. In one embodiment, information regarding whether the conversion is omitted (or whether the conversion is performed) may be transmitted to the decoding unit (1950) via a bitstream. The entropy encoding unit (1925) encodes and converts the quantized conversion coefficients and outputs them as a bitstream. The quantized conversion coefficients may be restored to spatial domain data through the inverse quantization and inverse conversion unit (1930). The inverse quantization and inverse conversion unit (1930) may perform inverse quantization on the quantized conversion coefficients and determine residual data by applying a conversion kernel. In one embodiment, the inverse quantization and inverse conversion unit (1930) may not perform inverse conversion.For example, if the transformation for residual data is omitted, the inverse quantization and inverse transformation unit (1930) may not perform the inverse transformation. The inverse quantization and inverse transformation unit (1930) can determine residual data by performing inverse quantization on the quantized transformation coefficients. The data in the restored spatial domain is output as a restored image after passing through the deblocking filtering unit (1935) and the loop filtering unit (1940). The restored image can be used as a reference image for the next input image after passing through the predictive encoding unit (1915).

[0232] The encoded image data among the bitstreams received by the decoding unit (1950) is restored into spatial domain residual data through the entropy decoding unit (1955) and the inverse quantization and inverse transform unit (1960). The predicted data and residual data output from the prediction decoding unit (1975) are combined to form spatial domain image data, and the deblocking filtering unit (1965) and the loop filtering unit (1970) can perform filtering on the spatial domain image data to output a restored image for the current original image. The restored image can be used as a reference image for the next original image by the prediction decoding unit (1975).

[0233] The loop filtering unit (1940) of the encoding unit (1910) performs loop filtering using filter information input according to user input or system settings. The filter information used by the loop filtering unit (1940) is output to the entropy encoding unit (1925) and transmitted to the decoding unit (1950) along with the encoded image data. The loop filtering unit (1970) of the decoding unit (1950) can perform loop filtering based on the filter information input from the decoding unit (1950).

[0234] Figure 20 is a conceptual diagram showing the motion vector coding method of a decoder according to the VVC (Versatile Video Coding) standard.

[0235] The motion vector MV encoding method (2000) performed in the decoder according to the VVC standard can be branched into a merge / skip method (2020) and an AMVP (advanced motion vector prediction) method (2060) depending on the value of the syntax element merge_flag (2010). When the value of merge_flag (2010) is 1, the operation of predicting the motion vector of the current block using the motion vector of surrounding blocks can be performed according to the merge / skip method (2020).

[0236] When the value of merge_flag (2010) is 0, the motion vector difference from the bitstream can be directly parsed according to the AMVP method (2060), and the motion vector difference can be combined with the motion vector of the surrounding block to predict the motion vector of the current block.

[0237] Specifically, in the merge / skip method (2020), depending on whether it is a sub-block-based inter prediction (2030), an extended merge candidate list (2040) or an affine merge candidate list (2050) may be used to select the motion vector of the surrounding blocks.

[0238] If inter prediction based on subblocks (2030) is not performed, inter prediction according to the merge mode can be performed using an extended merge candidate list (2040). Multiple candidates that can be included in the extended merge candidate list (2040) may be spatial MVP candidates, temporal MVP candidates, HMVP (History-based MVP) candidates, PMVP (Pairwise average MVP) candidates, or zero MV candidates. HMVP candidates are MVs of already restored encoding units and may mainly be MVs from non-adjacent encoding units. HMVP candidates are stored in a table of 5 entries and can be updated according to a FIFO (first-in-first-out) rule. PMVP is generated by averaging the MV of the first candidate and the MV of the second candidate of the merge candidate list for each reference picture list, and an average can be generated even if the two candidates have different reference images. The reference image index of the first candidate may be assigned to the reference image index of the PMVP. If only one of the two candidate MVs is available, the available MV may be used as the PMVP. Additionally, if there are no unavailable MVs for a specific reference video list, the MVs in that list may be deemed invalid.

[0239] When an extended merge candidate list (2040) is used, motion vectors can be predicted according to a general merge mode, an MMVD mode (merge mode with motion vector difference), and a GPM mode (geometrical partitioning mode). In particular, in the case of the MMVD mode, an MMVD candidate list (MMVD candidate list, 2045) is constructed, and motion vectors can be predicted by combining simplified motion vector differences with selected candidates from the MMVD candidate list.

[0240] In the case of subblock-based inter prediction (2030), inter prediction according to the affine merge mode can be performed using the affine merge candidate list (2050). The affine merge mode is one of the subblock-based merge modes. Affine merge candidates, i.e., CPMV (control point motion vector) candidates, can be added to the subblock-based merge candidate list. According to the affine merge mode, motion compensation prediction based on an affine motion model using CPMV derived from the subblock-based merge candidate list can be performed. In the affine merge mode, there are two types of CPMV candidates: inherited affine merge candidates and constructed affine merge candidates.

[0241] The inherited affine merge candidate may be the CPMV of the current encoding unit derived from the CPMV of the upper or left neighbor encoding unit, where motion compensation is performed based on the affine motion model. The constructed affine merge candidate may be constructed by combining translation motion information of neighboring CUs corresponding to the control point.

[0242] Specifically, in the motion vector prediction method according to the AMVP method (2060), an extended AMVP candidate list (2080) or an affine AMVP candidate list (2090) may be used to select the motion vector of a surrounding block depending on whether it is a sub-block-based inter prediction (2070).

[0243] If inter prediction based on subblocks (2070) is not performed, inter prediction according to the General AMVP mode can be performed using the extended AMVP candidate list (2080). Candidates that can be included in the extended AMVP candidate list (2080) may be spatial MVP candidates, temporal MVP candidates, HMVP candidates, or zero MV candidates. For each reference list 0 and reference list 1, motion information including reference picture indices, MVP indices, and MVDs can be parsed from the bitstream.

[0244] In the case of subblock-based inter prediction (2070), inter prediction according to the affine AMVP mode (General AMVP mode) can be performed using the affine AMVP candidate list (2090). The affine inter mode is a type of AMVP mode, and the motion model type can be determined as a 4-parameter model or a 6-parameter model depending on the affine type flag. If the motion model type is a 4-parameter model, MVD information between two CPMVs and two predicted values ​​from the affine MVP list can be signaled. If the motion model type is a 6-parameter model, MVD information between three CPMVs and three predicted values ​​from the affine MVP list can be signaled.

[0245] The affine AMVP list may contain up to two affine AMVP candidates, which can be divided into inherited affine AMVP candidates and constructed affine AMVP candidates. Inherited affine AMVP candidates may be derived from CPMVs of upper or left coding units that have the same reference image as the current coding unit. Constructed affine AMVP candidates may be derived by combining translational motion information of neighboring coding units that have the same reference image as the current coding unit. Additionally, if there are fewer than two affine AMVP candidates, translational MVs of the same neighboring coding unit may be assigned to two or three CPMVs within the affine AMVP list, depending on the motion model type (whether it is 4-parameter or 6-parameter).

[0246] When a prediction sample is determined using motion information obtained through inter-prediction, a reconstructed sample can be obtained by combining the prediction sample with the residual sample. Generally, if residual components exist in the bitstream, residual samples can be decoded from the residual components parsed from the bitstream. However, if the residual components that need to be signaled through the bitstream can be reduced, the amount of data transmitted can be reduced.

[0247] Below, CCLM (Cross-Component Linear Model), CCCM (Convolutional Cross-Component Model), and LIC (Local Illumination Compensation) techniques for modeling samples are described in detail through FIGS. 21 to 23.

[0248] Figure 21 is a diagram illustrating a Cross-Component Linear Model (CCLM) for predicting chroma blocks.

[0249] According to the CCLM technique, the difference between the surrounding template (2110) of the luminance block (2100) and the surrounding template (2140) of the chroma block (2130) is modeled (2150). When the model for the difference between the two surrounding templates (2110, 2140) is applied to the luminance block (2110), a predicted sample value of the chroma block (2130) can be generated (2160) from the sample value of the luminance block (2110).

[0250] In the CCLM technique, a linear model of the form y=ax+b is used. a and b are coefficients determined through modeling the difference between the surrounding template (2110) and the surrounding template (2140) of the chroma block (2130), x is a luminance sample of the luminance block (2110), and y is a chroma prediction sample corresponding to the luminance sample position in the chroma block (2130). That is, the encoding unit can encode the residual component of the chroma prediction sample obtained through the CCLM and the actual chroma sample value. The decoding unit can recover the chroma sample by combining the residual component obtained from the bitstream with the chroma prediction sample obtained through the CCLM.

[0251] Figure 22 is a diagram illustrating the CCCM (Convolutional Cross-Component Model) for predicting chroma blocks.

[0252] According to the CCCM technique, unlike the CCLM technique of Fig. 21, a convolution model is used instead of a leading model. That is, a convolution filter is modeled to determine chroma prediction samples from luminance samples.

[0253] Taking the Luma Block (2200) as an example, the model according to CCCM is as shown in Equation 1.

[0254] [Equation 1]

[0255] P = C*C + (1 << (b - 1)) ) >> bitdepth,

[0256] Bias =1 << (b - 1), b = bitdepth

[0257] pred = c0C + c1A + c2B + c3L + c4R + c5P + c6Bias

[0258] Here, C is a chroma sample of the current position (2210); bitdepth is the bit depth (e.g., log2(256)= 8); pred is a chroma prediction sample corresponding to the current position (2210).

[0259] A, B, L, and R are the Luma sample at the left position (2240) of the current position (2210), the Luma sample at the right position (2250), the Luma sample at the upper position (2220), and the Luma sample at the lower position (2230), respectively.

[0260] By utilizing the difference between the surrounding template of the luminance block (2200) and the surrounding template of the chroma block, filter coefficients c0, c1, c2, c3, c4, c5, and c6 of the convolution filter can be determined. That is, as shown in Equation 1, by applying a convolution filter to the luminance sample C of the current position (2210), the luminance sample A of the left position (2240), the luminance sample B of the right position (2250), the luminance sample L of the upper position (2220), the luminance sample R of the lower position (2230), the nonlinear term P, and the bias, a chroma prediction sample pred can be obtained.

[0261] In the encoding unit, the residual component between the chroma prediction sample obtained through CCCM and the actual chroma sample value can be encoded. In the decoding unit, the chroma sample can be reconstructed by combining the residual component obtained from the bitstream with the chroma prediction sample obtained through CCCM.

[0262] Figure 23 is a diagram illustrating the Local Illumination Compensation (LIC) technology.

[0263] According to the LIC technique, a model for the difference between two templates (2310, 2340) is determined by using the surrounding template (2310) of the current block (2300) and the surrounding template (2340) of the reference block (2330). By applying the model to the reference block (2330), a predicted block of the current block (2300) can be obtained from the reference block (2330). In the LIC technique, a linear model of the form y=ax+b is also used.

[0264] The CCLM and CCCM techniques described above are sample modeling methods for predicting chroma samples from luminance samples, and the LIC technique is a sample value modeling method for predicting luminance samples from reference samples. The present disclosure first provides a method for constructing a residual model between a predicted sample and a reconstructed sample of a decoded block.

[0265] FIG. 24 is a block diagram of a video decoding device according to one embodiment.

[0266] Referring to FIG. 24, the video decoding device (2400) may include a processor (2410) and a memory (2420).

[0267] In one embodiment of the present disclosure, the processor (2410) may include processing circuits and / or multiple processors. For example, the processor (2410) may include various processing circuits including at least one processor, and at least one of the processors may be configured to perform the various functions described in the present disclosure individually and / or collectively in a distributed manner.

[0268] In one embodiment of the present disclosure, the memory (2420) may include one or more storage media that store at least one instruction. A processor (2410) may control the video decoder (2400) by executing the instruction stored in the memory (2420). For example, a processor (2310) may control the video decoder (2400) to perform an operation by executing the instruction stored in the memory (2420) individually or collectively. In one embodiment of the present disclosure, the operation performed by the video decoder (2400) may be an operation performed by the processor (2410) of the video decoder (2400).

[0269] In one embodiment of the present disclosure, the video decoding device (2400) may correspond to the image decoding device (100) shown in FIG. 1 and / or the decoding unit (1950) shown in FIG. 19.

[0270] The video decoding device (2400) can obtain a bitstream generated as a result of encoding by the video encoding device (3000). The bitstream may include the encoding result for the current block. The configuration and operation of the video encoding device (3000) will be described later through FIGS. 30 and 31.

[0271] In one embodiment of the present disclosure, a video decoder (2400) may receive a bitstream from a video encoding device (3000) via a network. In one embodiment of the present disclosure, the video decoder (2400) may obtain a bitstream from a data storage medium comprising at least one of a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, or a magneto-optical medium such as a floptical disk.

[0272] A video decoding device (2400) can obtain syntax elements for decoding an image from a bitstream. Values ​​corresponding to the syntax elements may be included in the bitstream according to the hierarchical structure of the image. In one embodiment of the present disclosure, the video decoding device (2400) can obtain empty strings by entropy decoding bit sequences included in the bitstream, and obtain syntax elements corresponding to the empty strings.

[0273] In one embodiment of the present disclosure, the bitstream may include information regarding the prediction mode of the current block within the current image. The current block may include at least one of a maximum encoding unit, an encoding unit, a transform unit, or a prediction unit divided from the current image to be decoded. In one embodiment of the present disclosure, the prediction mode of the current block may include at least one of an intra mode, an inter mode, a combined mode, a normalized merge mode, an affine merge mode, a normalized AMVP mode, an affine AMVP mode, a geometric partitioning mode, and a block copy mode.

[0274] The video decoder (2400) can determine a prediction mode based on information regarding the prediction mode of the current block. The video decoder (2400) can generate a prediction block of the current block by performing a prediction according to the prediction mode.

[0275] A video decoder (2400) according to one embodiment may determine a prediction sample of a encoding unit. Alternatively, when performing inter-prediction based on sub-blocks, the video decoder (2400) may determine a prediction sample of a encoding unit for each of two or more sub-partitions generated by dividing the encoding unit. After determining the prediction samples for each sub-partition, the video decoder (2400) may collect the prediction samples of the sub-partitions to determine a prediction block of the encoding unit.

[0276] The video decoder (2400) can generate a restored current block using a prediction block. In one embodiment of the present disclosure, the video decoder (2400) can determine the prediction block as the restored current block. In one embodiment of the present disclosure, the video decoder (2400) can generate a restored current block by combining the prediction block with residual data obtained from the bitstream by the video decoder (2300). The restored current block can be used as a reference block for the next block.

[0277] In one embodiment, the video decoder (2400) may obtain residual data from a bitstream. The residual data may include information regarding the difference between the original image (or original sample) and the predicted image (or predicted sample). In one embodiment, the video decoder (2400) may obtain the transformation coefficients of a residual block corresponding to a transformation unit from the bitstream. In one embodiment, the video decoder (2400) may obtain residual samples of a residual block based on the transformation coefficients of a residual block. For example, the video decoder (2400) may obtain residual samples of a residual block by performing at least one of inverse quantization or inverse transformation on the transformation coefficients of a residual block. The video decoder (2400) may restore samples of a encoding unit based on residual samples of an encoding unit.

[0278] A video decoding device (2400) according to one embodiment may first construct a residual model between a predicted sample and a restored sample of a decoded block so as not to directly signal a residual component encoded with residual sample values ​​through a bitstream. The video decoding device (2400) may obtain a restored sample from a predicted sample of a current block by applying the residual model to the prediction result of the current block.

[0279] In one embodiment, the video decoding device (2400) can determine that a second block is used for inter-predicting of a first block based on inter-predicting information of a first block obtained from a bitstream.

[0280] For example, the inter prediction information of the first block may be information pointing to the second block among the surrounding blocks used for inter prediction of the first block. The surrounding blocks used for inter prediction of the first block may be candidate blocks included in the candidate list described above through FIG. 20. For example, any one of the candidate lists may be used: the extended merge candidate list (2040) in the merge / skip method (2020), the MMVD candidate list (2045) in the MMVD mode, the affine merge candidate list (2050) in the affine merge mode, the extended AMVP candidate list (2080) in the General AMVP mode, and the affine AMVP candidate list (2090) in the affine AMVP mode.

[0281] For example, a video decoding device (2400) can obtain candidate information from a bitstream that points to a candidate included in a candidate list of a first block. Among the candidate blocks included in the candidate list, the candidate block pointed to by the candidate information can be determined as the second block. For example, if the inter-prediction mode of the first block is skip mode or merge mode, a candidate index representing the second block among the candidates included in the merge candidate list can be obtained. If the inter-prediction mode of the first block is AMVP mode, a candidate index representing the second block among the candidates included in the AMVP candidate list can be obtained. If the inter-prediction mode of the first block is affine mode, a candidate index representing the second block among the candidates included in the affine candidate list can be obtained. If the inter-prediction mode of the first block is mmvd mode, a candidate index representing the second block among the candidates included in the candidate list can be obtained. The second block can be determined based on the obtained candidate index.

[0282] A video decoding device (2400) according to one embodiment can identify whether a residual model of a second block can be used for inter-prediction of a first block.

[0283] For example, the video decoding device (2400) can determine whether the residual model of the second block can be used by using at least one of the skip mode flag, merge mode flag, or CBF (Coded block flag) of the second block.

[0284] The skip mode flag of the second block may be information indicating whether the inter-prediction of the second block is performed in skip mode. The merge mode flag of the second block may be information indicating whether the inter-prediction of the second block is performed in merge mode. The CBF of the second block may be information indicating whether a non-zero transformation factor exists in the second block. For example,

[0285] For example, the video decoder (2400) can determine through the skip mode flag of the second block that if it is not in skip mode, residual samples of the second block exist. Therefore, since it is expected that residual models of the second block exist, it can be determined that residual models of the second block are available. If it is identified that it is in skip mode through the skip mode flag of the second block, residual samples of the second block do not exist, so it can be determined that residual models of the second block are not available.

[0286] For example, if the second block is not in skip mode, the merge mode flag of the second block can be obtained. The video decoder (2400) can determine through the merge mode flag of the second block that if it is not in merge mode, residual samples of the second block exist. Therefore, since it is expected that a residual model of the second block exists, it can be determined that the residual model of the second block is available. If it is identified that it is in merge mode through the merge mode flag of the second block, residual samples of the second block may or may not exist, so the availability of the residual model can be determined by additionally looking at the CBF of the second block.

[0287] For example, if the video decoder (2400) identifies through the CBF of the second block that there is a non-zero transformation factor, it can expect that a residual model of the second block exists. Thus, it can determine that the residual model of the second block is available. If it identifies through the CBF of the second block that there is no non-zero transformation factor, it can determine that the residual model of the second block is not available.

[0288] A video decoding device (2400) according to one embodiment can obtain information from a bitstream indicating whether a residual model of a restored block can be used for each predetermined data unit.

[0289] For example, a video decoder (2400) can obtain information from a bitstream indicating whether a residual model of a second block can be used for inter-prediction of a first block for each picture, slice, coding tree unit (CTU), or block containing a first block.

[0290] For example, information indicating whether a residual model of a reconstructed block can be used from the picture parameter set (PPS) for the current picture can be obtained. If a residual model is available based on the obtained information, a residual model of another reconstructed block can be used to predict the inter-blocks included in the current picture.

[0291] For example, information indicating whether a residual model of a restored block is available can be obtained from the slice header for the current slice. If a residual model is available based on the obtained information, a residual model of another restored block can be used for inter-prediction of the blocks included in the current slice.

[0292] For example, information indicating whether a residual model of a restored block is available can be obtained from the NAL unit of the coding tree unit syntax for the current CTU. If a residual model is available based on the obtained information, a residual model of another restored block can be used for inter-prediction of the blocks included in the current CTU.

[0293] A video decoding device (2400) according to one embodiment may not generate a residual model of the second block if the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold. Accordingly, the video decoding device (2400) may identify that the residual model of the second block is not available if the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold.

[0294] A video decoding device (2400) according to one embodiment can generate a residual model of a second block when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold. Accordingly, the video decoding device (2400) can identify that the residual model of the second block is available when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold.

[0295] A video decoding device (2400) according to one embodiment can generate a residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block when the residual model of the second block can be used.

[0296] A video decoding device (2400) according to one embodiment can determine a recovery block of the first block by applying a residual model of the second block to a prediction block of the first block. The residual model of the second block can express the relationship between the prediction block of the second block and the recovery block. Therefore, by applying the residual model of the second block to the prediction block of the first block, a recovery block of the first block corresponding thereto can be determined.

[0297] A video decoding device (2400) according to one embodiment can determine a prediction block of a first block using motion information of a first block. For example, the motion information of the first block may include motion vectors, prediction direction information, reference picture indices, etc. Based on the prediction direction information, a reference list can be determined using an L0 list, an L1 list, or both. A reference picture can be determined among the restored pictures included in the reference list using the reference picture index. A reference sample within the reference picture can be determined using motion vectors. A reference block is an area containing reference samples.

[0298] A video decoding device (2400) according to one embodiment can determine motion information of a first block using motion information of a second block. For example, in a merge mode, motion vector, predicted direction information, and reference picture index of a first block can be determined using motion vector, predicted direction information, and reference picture index of a second block.

[0299] A video decoding device (2400) according to one embodiment can determine a prediction block of the first block using a reference block pointed to by motion information of the first block, and determine a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

[0300] A video decoding device (2400) according to one embodiment can generate a filtered recovery block of the first block by determining a recovery block of the first block using a prediction block of the first block, and then applying a residual model of the second block to the recovery block of the first block for in-loop filtering on the recovery block of the first block.

[0301] A video decoding device (2400) according to one embodiment may apply a residual model in units of a CTU or a slice. Specifically, the video decoding device (2400) may generate filtered restored blocks of a CTU by applying the residual model of a second block to a first block on which it is decided to apply the residual model of a second block after all restored blocks of blocks included in a CTU including a first block have been generated.

[0302] Additionally, the video decoding device (2400) can generate filtered recovery blocks of a slice by applying the residual model of the second block to the first block on which it is decided to apply the residual model of the second block after all recovery blocks of the blocks included in the slice including the first block have been generated.

[0303] FIG. 25 is a diagram illustrating a method of using a residual model of spatially adjacent candidate blocks according to one embodiment.

[0304] The prediction block generated as an inter-prediction result of the current block (2510) can be determined from the reference block (2560) within the reference picture (2550). The residual component of the current block (2510) can be estimated through a residual model between the prediction block and the restored block of the previously restored neighbor block (2520), rather than being obtained from the bitstream.

[0305] At this time, the neighbor block (2520) may be a block spatially adjacent to the current block (2510). According to the motion information of the neighbor block (2520), the prediction block of the neighbor block (2520) may also be determined from the reference block (2570) included in the reference picture (2550). A residual model (2530) representing a relationship between the prediction sample of the neighbor block (2520)'s prediction block and the restoration sample of the neighbor block (2520) may be determined. The residual model may be a linear model or a convolutional model.

[0306] When the residual model (2530) of the neighbor block (2520) is applied (2540) to the prediction block of the current block (2510), the recovery samples of the current block (2510) can be determined from the prediction samples of the current block (2510).

[0307] According to one embodiment, the point in time at which the residual model (2530) of the neighbor block (2520) is applied (2540) to the prediction block of the current block (2510) may be before the restoration of the CTU or slice containing the current block (2510) is completed. In this case, restoration samples of the current block (2510) are generated by applying the residual model (2530), and in-loop filtering may be performed on the restoration samples.

[0308] According to one embodiment, the point in time at which the residual model (2530) of the neighbor block (2520) is applied (2540) to the prediction block of the current block (2510) may be after the restoration of the CTU or slice containing the current block (2510) is completed. In this case, the prediction block of the current block (2510) becomes the restoration block of the current block (2510) as is, and the residual model (2530) is applied to the restoration block of the current block (2510) as a loop filtering process, thereby generating filtered restoration samples.

[0309] A video decoding device (2400) according to one embodiment may configure and store a residual model between a prediction block and a restoration block when there is a residual between a prediction block and a restoration block of a second block in an inter-prediction mode. When the second block is used for inter-prediction of a first block that is restored later than the second block, the video decoding device (2400) may apply the residual model of the second block to the first block.

[0310] In order for the video decoder (2400) to use the residual model of the second block after generating the prediction block of the first block, the recovery block of the second block must be determined, and a residual model based on the prediction block and the recovery block of the second block must already be configured. If the second block has a residual component, the video decoder (2400) can generate the recovery block of the first block by applying the residual model of the second block to the prediction block of the first block. On the other hand, if the first block does not have a residual component, the same recovery block as the prediction block of the first block is determined, so the video decoder (2400) can generate a filtered recovery block of the first block by applying the residual model of the second block to the recovery block of the first block. That is, the residual model of the second block can be applied as a filtering process of the first block.

[0311] When the residual model of the second block is applied as a filtering process of the first block and the first block is referenced for intra prediction of the third block, the restored block before filtering of the first block can be used for intra prediction. Accordingly, the restored block to which the residual model is not applied can be referenced for intra prediction of the third block.

[0312] As described above, depending on the process of checking whether the reference block contains residual information such as a residual model for the prediction of the current block, it is determined whether the residual signal of the current block needs to be decoded. Therefore, a problem may arise where the restoration process of the reference block must be waited for during the parsing stage of the current block. To solve this problem, the video decoding device (2400) may additionally obtain information from the bitstream indicating whether the residual model of the surrounding block is applied to the prediction block of the current block. If information regarding whether the residual model is applied is obtained, the motion information used for the prediction of the current block may be limited to motion information containing residual information. Therefore, in order to determine the availability of the residual model of the surrounding block, it can be identified whether the surrounding block contains residual information and has available motion information.

[0313] According to one embodiment, if there is a residual component between a prediction block and a restoration block determined by inter-prediction, the video decoder (2400) can construct a model for residual samples between the prediction block and the restoration block. At this time, various relational models can be utilized without being limited to specific models such as linear models or convolutional models.

[0314] A video decoding device (2400) according to one embodiment can identify whether there is a residual component between a prediction block and a recovery block in the current block through information obtained from a bitstream. For example, whether there is a residual component in the current block can be identified using a skip mode flag, a merge mode flag, a CBF (coded block flag), etc. obtained from the bitstream.

[0315] For example, if the CBF of the current block obtained from the bitstream by the video decoder (2400) indicates 1, it may obtain residual information related to the current block from the bitstream and restore residual samples using the residual information. The video decoder (2400) may determine restored samples using residual samples and prediction samples. The video decoder (2400) may determine a residual model using the difference between the restored samples and prediction samples of the current block. A relationship of a linear model or a convolutional model for obtaining restored samples from prediction samples may be determined.

[0316] A video decoding device (2400) according to one embodiment may determine whether to generate a residual model by using the size of residual samples to reduce the amount of computation required to determine the residual model. For example, even if there are residual components between the prediction samples and the reconstructed samples of the current block, if the sum of all residual samples is not greater than a predetermined threshold, a residual model for the current block may not be generated.

[0317] Information on whether a residual model can be applied to a prediction block can be obtained for each block, CTU, slice, and picture.

[0318] For example, when information regarding whether a residual model can be applied to a prediction block is obtained from a NAL unit corresponding to the coding unit syntax of the current block, the obtained information can be used to determine whether a residual model can be applied to a prediction block of the current block.

[0319] For example, when information regarding whether a residual model can be applied to a prediction block is obtained from a NAL unit corresponding to the coding tree syntax of a CTU, the obtained information can be used to determine whether a residual model can be applied to the prediction blocks of the blocks included in the current CTU.

[0320] For example, when information regarding whether a residual model can be applied to a prediction block is obtained from a NAL unit corresponding to a slice header, the obtained information can be used to determine whether a residual model can be applied to the prediction blocks of the blocks included in the current slice.

[0321] For example, when information regarding whether a residual model can be applied to a prediction block is obtained from a NAL unit corresponding to a PPS, the obtained information can be used to determine whether a residual model can be applied to the prediction blocks of the blocks included in the current picture.

[0322] When a residual model is applied, only neighboring blocks possessing a residual model can be available candidate blocks. Therefore, since the candidate list containing neighboring blocks used for inter-prediction includes only available candidate blocks, it can only include neighboring blocks for which motion information containing a residual model has been determined. Neighboring blocks for which a residual model has not been determined cannot be candidate blocks.

[0323] If a surrounding block for which a residual model has not been determined becomes an available candidate block, the video decoder (2400) does not separately acquire residual information of the surrounding block. The video decoder (2400) can acquire motion information of the current block and determine the predicted block using only motion information without a residual model of the surrounding block.

[0324] According to one embodiment, when using a surrounding block having motion information including a residual model, the video decoder (2400) can determine the motion information of the current block using the motion information of the surrounding block. The prediction block can be determined using a reference block pointed to by the motion information of the current block. The video decoder (2400) can use the prediction block of the current block as is as the restoration block of the current block without the need to additionally obtain residual information of the current block from the bitstream. The restoration block may also be updated by applying the residual model included in the motion information of the surrounding block to the restoration block of the current block.

[0325] When using a residual model without acquiring residual information of the current block, the residual model may be applied during the filtering step of the current block. According to one embodiment, the residual model may be applied first before loop filtering is applied to the restored block. According to one embodiment, the residual model may be linked to a specific filtering method, such as an adaptive loop filter (ALF), so that the loop filtering and the application of the residual model are performed simultaneously.

[0326] A video decoding device (2400) according to one embodiment can parse residual components of a portion of a block from a bitstream and generate a restored sample of the portion of the portion restored from the parsed residual components. The video decoding device (2400) can generate a residual model using the residual components of the portion of the portion, the restored sample, and the predicted sample. In this case, the residual model may be a linear model or a convolutional model.

[0327] The video decoder (2400) can generate only prediction samples for the remaining regions, excluding the regions where residual components are parsed from the bitstream, without obtaining residual components for the remaining regions. The video decoder (2400) can generate restoration samples for the remaining regions by applying the residual model generated from the regions to the prediction samples for the remaining regions.

[0328] Referring to FIG. 26 below, a method for applying a residual model generated from a part of a block to the remaining area is described.

[0329] FIG. 26 is a diagram illustrating a method for predicting residual information of a portion of a block according to one embodiment.

[0330] Encoding units are illustrated in FIG. 26. In each encoding unit, only the residual component for the dark region can be obtained from the bitstream. Accordingly, the video decoder (2400) can reconstruct the residual sample of the dark region using the residual component obtained from the bitstream. The video decoder (2400) can obtain the reconstructed samples by combining the reconstructed residual sample with the predicted sample of the dark region. Additionally, the video decoder (2400) can generate a residual model based on the predicted samples of the dark region and the reconstructed samples.

[0331] The video decoder (2400) can determine predicted samples of bright regions. However, the residual component of the bright region is not obtained from the bitstream. However, the video decoder (2400) can obtain restored samples of the bright region by applying a residual model generated from the dark region to the predicted samples of the bright region.

[0332] As a specific example, only the residual component for the dark area (2601) can be obtained from the bitstream without the residual component for the bright area (2651). Accordingly, the video decoder (2400) can reconstruct residual samples of the dark area (2601) using the residual component obtained from the bitstream. Reconstructed samples of the dark area (2601) can be obtained by combining the reconstructed residual samples with the predicted samples of the dark area (2601). Additionally, the video decoder (2400) can determine a residual model based on the difference between the predicted samples and the reconstructed samples of the dark area (2601).

[0333] The video decoder (2400) can determine predicted samples of the bright area (2651). However, the video decoder (2400) can obtain restored samples of the bright area (2651) by applying a residual model generated from the dark area to the predicted samples of the bright area (2651).

[0334] Similarly, the reconstructed samples of the bright area (2652) can be determined by applying the residual model obtained from the predicted samples and reconstructed samples of the dark area (2602) to the predicted samples of the bright area (2652). The reconstructed samples of the bright area (2653) can be determined by applying the residual model obtained from the predicted samples and reconstructed samples of the dark area (2603) to the predicted samples of the bright area (2653). The reconstructed samples of the bright area (2654) can be determined by applying the residual model of the dark area (2604) to the predicted samples of the bright area (2654). The reconstructed samples of the bright area (2655) can be determined by applying the residual model of the dark area (2605) to the predicted samples of the bright area (2655). The reconstructed samples of the bright area (2656) can be determined by applying the residual model of the dark area (2606) to the predicted samples of the bright area (2656). Residual models of the dark area (2607) can be applied to the bright area (2657) to determine the reconstructed samples of the bright area (2657), and residual models of the dark area (2608) can be applied to the bright area (2658) to determine the reconstructed samples of the bright area (2658). Residual models of the dark area (2609) can be applied to the bright area (2659) to determine the reconstructed samples of the bright area (2659), and residual models of the dark area (2660) can be applied to the bright area (2610) to determine the reconstructed samples of the bright area (2610). Residual models of the dark area (2611) can be applied to the bright area (2661) to determine the reconstructed samples of the bright area (2661), and residual models of the dark area (2612) can be applied to the bright area (2662) to determine the reconstructed samples of the bright area (2662).

[0335] Additionally, the video decoder (2400) can generate a residual model by obtaining residual components from the bitstream only in dark regions (2600) spaced apart at a predetermined interval within a block, such as blocks (2620, 2630, 2640, 2670). Since residual components for the bright regions (2650) of the blocks (2620, 2630, 2640, 2670) are not obtained from the bitstream, the video decoder (2400) can determine the restored samples of the bright regions (2650) by applying the residual model of the dark regions (2600) to the predicted samples of the bright regions (2650).

[0336] For regions where residual components are obtained from the bitstream, residual samples can be recovered by performing transform / inverse transform and quantization / inverse quantization. If residual components can be obtained in various forms of regions, such as the blocks exemplified in FIG. 26, information indicating which form of region allows for the acquisition of residual components can be additionally obtained from the bitstream. Alternatively, information regarding locations within the block where residual components can be obtained can be additionally obtained from the bitstream.

[0337] Accordingly, the video decoder (2400) can obtain residual components from the bitstream. Residual components can be obtained only for some regions within the block. The video decoder (2400) can determine the location of the regions where residual components are obtained by obtaining information about the regions or locations for obtaining residual components from the bitstream.

[0338] The video decoder (2400) can restore residual samples of some regions using residual components parsed from the bitstream. The restored samples of some regions can be determined using the restored residual samples and prediction samples. Additionally, a residual model can be generated based on the difference between the prediction samples and the restored samples of some regions.

[0339] The video decoder (2400) can predict restored samples by applying a residual model to the remaining area where residual components are not acquired. The restored samples of the remaining area can be predicted by applying a residual model to the predicted samples of the remaining area where residual components are not acquired.

[0340] Below, the method for generating and applying a residual model in the loop filtering step is described. Figure 27 illustrates the method of using residual components to generate a residual model.

[0341] FIG. 27 is a diagram illustrating a method for applying a residual model of some CUs within a CTU to the remaining CUs that have no residual components for loop filtering according to one embodiment.

[0342] The CTU (2700) includes encoding units (2712, 2714, 2716, 2718, 2752, 2754, 2756). For the encoding units (2712, 2714, 2716, 2718) which are dark regions, residual components can be parsed from the bitstream. Thus, residual samples for the encoding units (2712, 2714, 2716, 2718) can be recovered, and recovered samples can be generated using the predicted samples and the residual samples. On the other hand, for the encoding units (2752, 2754, 2756) which are bright regions, residual components are not parsed from the bitstream.

[0343] A video decoder (2400) according to one embodiment can generate a residual model between predicted samples and restored samples in encoding units (2712, 2714, 2716, 2718). The video decoder (2400) can apply the residual model generated from encoding units (2712, 2714, 2716, 2718) to encoding units (2752, 2754, 2756) which are bright areas. In this case, the residual model may be a linear model or a convolutional model.

[0344] The video decoder (2400) can generate a residual model for some areas and apply the residual model to the remaining areas within blocks of sizes 256x256, 128x128, 64x64, 32x32, and 16x16, as well as the CTU (2700). Additionally, information regarding the areas where the residual model is generated, such as the location, size, and range of the areas where residual components exist, can be obtained from the bitstream. Furthermore, information regarding whether to generate a residual model for each area, and information regarding whether to apply the residual model to the remaining areas, can be obtained from the bitstream. The information regarding whether a residual model is generated in the current area may be information regarding whether a residual model is generated based on the difference between the predicted sample and the restored sample, or information regarding whether it can be generated, since the residual component for the current area is obtained from the bitstream. Information regarding whether a residual model is applied to the current region may be information regarding whether a residual model from another region is applied to the prediction samples of the current region, or whether it can be applied, since the residual component for the current region was not obtained from the bitstream.

[0345] For example, the video decoding device (2400) selects an area to which a filter is to be applied, such as a picture, frame, slice, CTU, 256x256, 128x128, 64x64, 32x32, 16x16, etc. That is, the area to which a residual model is to be generated or the area to which a residual model is to be applied can be determined for each predetermined data unit.

[0346] At this time, information about the area where the residual model is to be generated or information about the area where the residual model is to be applied may be transmitted from the upper data unit. For example, if the residual model is to be applied per CTU, information on whether it can be applied per CTU is obtained from the bitstream per slice, frame, or picture, and whether the residual model will be applied to all CTUs included in the slice, frame, or picture may be determined.

[0347] In addition, whether to apply a filter to individual region levels can be determined separately. For example, when the CTU is 128x128, information regarding whether to apply a residual model for every 64x64 blocks can be obtained from the bitstream. For example, information regarding whether to apply a residual model for every 64x64 blocks can be obtained sequentially in the front part of the NAL unit corresponding to the CTU. Accordingly, the video decoder (2400) can apply a residual model to the encoding units among the encoding units where the residual model can be applied, based on the information regarding whether to apply a residual model for each encoding unit included in the CTU obtained from the bitstream.

[0348] In the picture parameter set or slice header, information regarding whether to apply a residual model to the encoding units included in the picture or slice can be obtained sequentially. Accordingly, the video decoder (2400) can apply a residual model to the encoding units to which the residual model can be applied, based on the information regarding whether to apply a residual model to the encoding units included in the picture or slice.

[0349] Additionally, information regarding whether to apply a residual model for each encoding unit can be obtained from the bitstream. Accordingly, based on the information regarding whether to apply a residual model for the current encoding unit obtained from the bitstream, the video decoder (2400) can apply a residual model to the current encoding unit if it is possible to apply the residual model.

[0350] The video decoder (2400) does not need to parse residual components from the bitstream for encoding units to which the residual model can be applied. Therefore, since the amount of computation and data for decoding residual components is reduced, more efficient decoding is possible.

[0351] In addition, since information regarding the region where a residual model can be generated can also be obtained on a data-by-data basis, the determination of whether to generate a residual model can be made on a data-by-data basis.

[0352] A video decoding device (2400) according to one embodiment can generate a residual model at a data unit level where the residual model can be applied. For example, if the residual model can be applied at the CTU level according to information regarding the area where the previously acquired residual model can be applied, the residual model can be generated at the CTU level. Therefore, it may be possible to generate a residual model for all encoding units included in the CTU.

[0353] If it is possible to apply a residual model at the slice level based on the acquired information, it may also be possible to generate a residual model at the slice level. In this case, it may be possible to generate a residual model for all coding units included in the slice.

[0354] If it is possible to apply a residual model at the picture level based on the acquired information, it may also be possible to generate a residual model at the picture level. In this case, it may be possible to generate a residual model for all encoding units included in the picture.

[0355] As another example, a residual model can be generated at a higher data unit level than the data unit to which the residual model can be applied.

[0356] For example, if a residual model is applicable to each CTU, a residual model can be generated for each slice, frame, or picture containing the CTU. That is, a residual model can be generated from all CTUs included in a slice, frame, or picture.

[0357] For example, based on information regarding whether to apply a residual model to each coding unit included in a CTU obtained from a bitstream, if there exists a coding unit to which a residual model can be applied, a residual model can be generated in the CTU, and a residual model can be generated in the coding unit included in the CTU.

[0358] In the picture parameter set or slice header, if a residual model can be applied to the encoding units included in the picture or slice based on whether the acquired residual model is applied, it may be possible to generate a residual model at the picture or slice unit.

[0359] In addition, even if information regarding the applicability of the residual model is not separately obtained from the bitstream, whether the residual model is applicable can be determined according to a predetermined rule.

[0360] Depending on rules or information regarding the applicability of the residual model, whether or not to apply the residual model can be determined if it is applicable.

[0361] Additionally, even if the application of a residual model to a data unit is possible (applicability) according to information or rules regarding whether the residual model can be applied, information regarding whether the residual model is applied to that data unit may be obtained separately from the bitstream. Alternatively, even if the application of a residual model to a data unit is possible (applicability), whether the residual model is applied to that data unit may be determined according to a predetermined rule.

[0362] In addition, even if information regarding the possibility of generating a residual model is not separately obtained from the bitstream, whether a residual model can be generated can be determined according to a predetermined rule.

[0363] Based on rules or information regarding the possibility of generating a residual model, whether or not to generate a residual model can be determined if it is possible.

[0364] Additionally, even if the generation of a residual model is possible (applicability) for a data unit according to information or rules regarding whether the generation of a residual model is possible, information regarding whether a residual model is generated for that data unit may be obtained separately from the bitstream. Alternatively, even if the generation of a residual model is possible (possibility of generation) for a data unit, whether a residual model is generated for that data unit may be determined according to a predetermined rule.

[0365] A video decoding device (2400) according to one embodiment may obtain, respectively, information regarding whether a residual model can be applied or information regarding whether a residual model is applied, information regarding whether a residual model can be generated or information regarding whether a residual model is generated from a bitstream. Based on the information regarding whether a residual model can be applied, the video decoding device (2400) may determine a data unit to which a residual model can be applied or determine whether a residual model can be applied in a current data unit. Based on the information regarding whether a residual model is applied, the video decoding device (2400) may determine a data unit to which a residual model is applied or determine whether a residual model is applied in a current data unit. Based on the information regarding whether a residual model can be generated, the video decoding device (2400) may determine a data unit to which a residual model can be generated or determine whether a residual model can be generated in a current data unit. The video decoder (2400) can determine the data unit for which the residual model is generated or determine whether the residual model is generated in the current data unit based on information regarding whether the residual model is generated.

[0366] A video decoding device (2400) according to one embodiment can generate a residual model for an area where residual information exists among the areas where a residual model can be generated. The residual model according to one embodiment may be a linear model or a convolutional model, etc. If residual information of the current encoding unit exists, the video decoding device (2400) can restore residual samples and determine restored samples of the current encoding unit by combining prediction samples and residual samples. Additionally, the video decoding device (2400) can generate a residual model based on the prediction samples and restored samples.

[0367] For example, if the video decoder (2400) generates prediction samples using two or more reference images, it can generate a final prediction sample using the first prediction samples obtained from the first reference image and the second prediction samples obtained from the second reference image. In this case, the video decoder (2400) can generate a residual model using the final prediction samples and the restored samples.

[0368] Additionally, the video decoding device (2400) can apply a residual model to each data unit subject to filtering. For example, even if the unit in which the residual model is generated is an encoding unit, if the target of filtering is a CTU unit, the residual model can be applied to the CTU unit.

[0369] A video decoding device (2400) according to one embodiment can apply a residual model to areas without residual information among data units to which filtering is applied.

[0370] For example, if the data unit to which filtering is applied is a CTU, a residual model is generated from encoding units within the CTU where residual information exists, and the residual model can be applied to encoding units within the CTU where residual information does not exist.

[0371] As another example, when the data unit to which filtering is applied is a CTU, a residual model can be generated from CTUs where residual information exists, and the residual model can be applied to CTUs where residual information does not exist.

[0372] As another example, the area where the residual model is applied can be the entire data unit to which filtering is applied, regardless of the presence or absence of residual information. Even if residual information exists, if only negligible residuals are present, the additional application of the residual model can be expected to complement the residual components.

[0373] According to one embodiment, a filter to which a residual model is applied can be implemented independently of other filters.

[0374] When a residual model filter is applied in parallel with other filters, the samples output through the residual model's filtering process can become the inputs to the other filters. Additionally, the weighted sum of the outputs of each filter, including the filter to which the residual model is applied and other filters, can be the final output of the filtering.

[0375] FIG. 28 is a diagram illustrating a method for generating a residual model between predicted regions and restored regions within a picture according to one embodiment and applying it to regions without residual information.

[0376] A video decoding device (2400) according to one embodiment may generate and apply a residual model of the entire picture.

[0377] Specifically, the video decoder (2400) obtains the residual components of the first encoding units (2810) of the dark area from the bitstream within the picture (2800), but for the bright area, the residual components of the second encoding units (2820) cannot be obtained from the bitstream.

[0378] Accordingly, the video decoder (2400) can obtain the recovery blocks (2850) of the first encoding units (2810) using the prediction blocks (2840) and residual components of the first encoding units (2810). The video decoder (2400) can generate a residual model to be applied to the picture (2800) using the prediction blocks (2840) and the recovery blocks (2850). That is, the residual model may also be generated using the first encoding units (2810), which are a plurality of encoding units included in the picture (2800).

[0379] The video decoder (2400) can obtain filtered samples (2880) of the second encoding units (2820) by applying a residual model generated for the picture (2800) to the second encoding units (2820). Through filtering applied to the prediction samples of the second encoding units (2820) by the residual model, the filtered samples (2880) can be output as restored samples of the second encoding units (2820).

[0380] Similar to the embodiment in FIG. 28 where a residual model is generated and applied at the picture level, the video decoder (2400) may generate and apply a residual model not only at the encoding unit level but also at the CTU and slice level. Specifically, instead of generating a residual model for each encoding unit included in the CTU, the video decoder (2400) generates reconstructed samples using residual samples and prediction samples from each encoding unit that has residual information among the encoding units included in the CTU. The video decoder (2400) may generate a single residual model using prediction samples and reconstructed samples from multiple encoding units, rather than generating a residual model for each encoding unit. In this case, the video decoder (2400) can obtain filtered reconstructed samples of each encoding unit that does not have residual information by applying the generated single residual model to the encoding units that do not have residual information among the encoding units included in the CTU.

[0381] It takes time until the point of restoration of the reference block for the residual model to be generated. When the residual model is applied block by block, if the point of restoration of the reference block is later than the point of completion of generation of the prediction block of the current block, the residual model has not yet been generated. Consequently, a problem may arise where the application of the residual model to the prediction block for the restoration of the current block is delayed. However, as shown in Fig. 28, when the residual model is applied to a data unit larger than the encoding unit, the target of the filtering to which the residual model is applied expands from the encoding unit level to the CTU unit, slice unit, or picture unit. Consequently, the timing of the filtering execution is also delayed to the point of restoration at the CTU unit, slice unit, or picture unit level. This delay in the execution of the filtering to which the residual model is applied has the advantage of securing time for the residual model to be generated.

[0382] FIG. 29 is a flowchart showing a video decoding method according to one embodiment.

[0383] In step 2910, the video decoding device (2400) can determine that the second block is used for inter-prediction of the first block based on the inter-prediction information of the first block obtained from the bitstream.

[0384] In step 2920, the video decoder (2400) can identify whether the residual model of the second block can be used for the inter-prediction of the first block.

[0385] A video decoding device (2400) according to one embodiment can determine whether a residual model of the second block can be used by using a skip mode flag indicating whether the inter-prediction of the second block is performed in skip mode. If the inter-prediction of the second block is performed in skip mode, a residual model is not generated because there are no residual components of the second block. Therefore, it can be determined that a residual model cannot be used.

[0386] The video decoder (2400) can determine whether the residual model of the second block can be used by using a merge mode flag indicating whether the inter prediction of the second block is performed in merge mode. If the inter prediction of the second block is performed in merge mode, there is a high probability that residual components exist, so the residual model of the second block can be generated and the residual model can be applied to the first block. The video decoder (2400) can determine whether the residual model of the second block can be used by using a CBF indicating whether there are non-zero transform coefficients in the second block. If it is determined based on the CBF that there are no non-zero transform coefficients in the second block, then there are no residual components in the second block, so the residual model is not generated. Therefore, it can be determined that the residual model cannot be used. If there are non-zero transform coefficients in the second block based on the CBF, then there are residual components, so the residual model of the second block can be generated and the residual model can be applied to the first block.

[0387] A video decoding device (2400) according to one embodiment may not generate a residual model of the second block if the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold. If the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold, a residual model of the second block may be generated.

[0388] A video decoding device (2400) according to one embodiment can obtain information from a bitstream indicating whether a residual model of a second block can be used for inter-prediction of a first block for each picture, slice, CTU, or block including a first block.

[0389] In step 2930, the video decoder (2400) can generate a residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block when the residual model of the second block is available.

[0390] In step 2940, the prediction block of the first block is determined using the motion information of the first block, and the restoration block of the first block can be determined by applying the residual model of the second block to the prediction block of the first block.

[0391] A video decoding device (2400) according to one embodiment can determine motion information of a first block using motion information of a second block. A prediction block of a first block can be determined using a reference block pointed to by motion information of the first block.

[0392] A video decoding device (2400) according to one embodiment can determine a recovery block of a first block using a prediction block of a first block, and generate a filtered recovery block of a first block by applying a residual model of a second block to the recovery block of a first block for in-loop filtering on the recovery block of a first block.

[0393] A video decoding device (2400) according to one embodiment can generate a filtered restoration block of the first block by applying a residual model of the second block as a filtering to a prediction block of the first block.

[0394] According to one embodiment, when the filtering application unit is a CTU or a slice, after the restoration blocks of the blocks included in the CTU or slice including the first block are generated, the video decoding device (2400) may apply the residual model of the second block to the first block on which it is determined to apply the residual model of the second block.

[0395] The method for generating and applying a residual model during the video decoding process was explained above. The method for generating and applying a residual model during the video process is described in detail below.

[0396] FIG. 30 is a block diagram of a video encoding device according to one embodiment.

[0397] A video encoding device (3000) according to one embodiment includes a memory (3020) for storing instructions and at least one processor (3010). At least one processor (3010) can be operably coupled to the memory (3020).

[0398] In one embodiment of the present disclosure, the processor (3010) may include processing circuits and / or multiple processors. For example, the processor (3010) may include various processing circuits including at least one processor, and at least one of the at least one processor may be configured to perform the various functions described in the present disclosure individually and / or collectively in a distributed manner.

[0399] In one embodiment of the present disclosure, the memory (3020) may include one or more storage media that store at least one instruction. The processor (3010) may control the video encoding device (3000) by executing the instruction stored in the memory (3020). For example, the processor (3010) may control the video encoding device (3000) to perform an operation by executing the instruction stored in the memory (3020) individually or collectively. In one embodiment of the present disclosure, the operation performed by the video encoding device (3000) may be an operation performed by the processor (3010) of the video encoding device (3000).

[0400] In one embodiment of the present disclosure, the video encoding device (3000) may correspond to the image encoding device (200) shown in FIG. 2 and / or the encoding unit (1910) shown in FIG. 19.

[0401] The video encoding device (3000) can determine the prediction mode of the current block. The current block may be a maximum encoding unit, encoding unit, conversion unit, or prediction unit divided from the current picture to be encoded.

[0402] In one embodiment, the prediction mode of the current block may be determined to be an intra mode or an inter mode. If it is an inter mode, the prediction mode may be specified in detail as a skip / merge mode, a regular merge mode, an MMVD mode, an affine merge mode, a GPM mode, a regular AMVP mode, an affine AMVP mode, etc.

[0403] In one embodiment, the video encoding device (3000) performs inter prediction for the current block according to the inter prediction mode of the current block, and can encode the current block using the prediction block generated as a result of the inter prediction.

[0404] In one embodiment, encoding of the current block may mean the process of generating information that enables the video encoding device (3000) to restore the current block. The information generated through encoding may be included in a bitstream.

[0405] In one embodiment, the video encoding device (3000) may generate residual samples corresponding to the difference between the predicted samples of the prediction block and the original samples of the current block. If the prediction block is determined to be the current block, residual data may not be generated. The video encoding device (3000) may generate quantized transformation coefficients by performing transformation and quantization on the residual samples, which are the difference between the predicted samples generated as an inter-prediction result and the residual samples. The video encoding device (3000) may encode encoding information corresponding to the encoding methods used to generate the quantized transformation coefficients. The video encoding device (3000) may generate bit sequences by performing entropy encoding on the transformation coefficients and encoding information, and generate a bitstream containing the bit sequences according to the order of the syntax elements.

[0406] By having at least one processor (3010) execute instructions either alone or in cooperation, the video encoding device (3000) can determine the motion vector of the current block through motion prediction.

[0407] A video encoding device (3000) according to one embodiment can identify whether a residual model of a second block can be used for inter-prediction of a first block. When a residual model of a second block can be used, the video encoding device (3000) can generate a residual model of a second block based on residual information between a prediction block of a second block and a restoration block of a second block. A residual model of a second block may also be generated based on the difference between a prediction block of a second block and a restoration block of a second block. The video encoding device (3000) can determine a prediction block of a first block and determine a restoration block of a first block by applying a residual model of a second block to a prediction block of a first block.

[0408] The video encoding device (3000) can determine that the second block is used for inter-prediction of the first block.

[0409] The video encoding device (3000) can encode information for determining that the surrounding block used for inter-predicting of the first block is the second block. The video encoding device (3000) can output a bitstream containing the encoded information for determining that the surrounding block used for inter-predicting of the first block is the second block. The output bitstream can be transmitted to another device or server or stored on a recording medium.

[0410] FIG. 31 is a flowchart showing a video encoding method according to one embodiment.

[0411] In step 3110, the video encoding device (3000) can identify whether the residual model of the second block can be used for the inter-prediction of the first block.

[0412] For example, the video encoding device (3000) can determine whether the residual model of the second block can be used based on whether the inter-prediction mode of the second block is a skip mode. If the inter-prediction of the second block is performed in a skip mode, the residual model of the second block cannot be used because there is no residual component in the second block.

[0413] For example, the video encoding device (3000) can determine whether the residual model of the second block can be used based on whether the inter prediction mode of the second block is a merge mode. If the inter prediction of the second block is not performed in a merge mode, the residual model of the second block can be used because there is a residual component in the second block. If the inter prediction of the second block is performed in a merge mode, the residual model of the second block can be used because there is a residual component in the second block.

[0414] The video encoding device (3000) can determine whether the residual model of the second block can be used based on whether there are non-zero transformation coefficients in the second block. If there are no non-zero transformation coefficients in the second block, the residual model of the second block does not exist, so the residual model cannot be used.

[0415] A video encoding device (3000) according to one embodiment may not generate a residual model of the second block if the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold value.

[0416] A video encoding device (3000) according to one embodiment can generate a residual model of a second block when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold.

[0417] Additionally, a video encoding device (3000) according to one embodiment may encode information indicating whether a residual model of a second block can be used for inter-prediction of a first block for each picture, slice, coding tree unit (CTU), or block including a first block.

[0418] In step 3120, when the video encoding device (3000) can use the residual model of the second block, it can generate the residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block. The residual model of the second block may also be generated based on the difference between the prediction block of the second block and the restoration block of the second block.

[0419] A residual model according to one embodiment may follow a linear model or a convolutional model.

[0420] In step 3130, the video encoding device (3000) can determine the prediction block of the first block and determine the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block.

[0421] A video encoding device (3000) according to one embodiment can determine motion information of a first block using motion information of a second block. When the inter-prediction mode of the first block is a skip mode, a merge mode, an AMVP mode, etc., motion information of the first block can be determined using motion information of the second block.

[0422] A video encoding device (3000) according to one embodiment can determine a prediction block of a first block using a reference block pointed to by motion information of a first block. A video encoding device (3000) according to one embodiment can determine a restoration block of a first block by applying a residual model of a second block to the prediction block of a first block.

[0423] A video encoding device (3000) according to one embodiment determines the prediction block of the first block as the restoration block of the first block, and may apply a residual model of the second block to the restoration block of the first block for in-loop filtering on the restoration block of the first block. By applying the residual model of the second block to the first block as in-loop filtering, a filtered restoration block of the first block can be generated.

[0424] Additionally, a video encoding device (3000) according to one embodiment may generate a filtered restoration block of the first block through a loop filtering process that directly applies a residual model of the second block to the prediction block of the first block.

[0425] A video encoding device (3000) according to one embodiment may apply the residual model of the second block to the first block on which it is decided to apply the residual model of the second block after the generation of restoration blocks of blocks included in a CTU or slice including the first block is completed.

[0426] In step 3140, the video encoding device (3000) can encode information to determine that the surrounding block used for inter-prediction of the first block is the second block.

[0427] For example, if the inter-prediction mode of the first block is skip mode or merge mode, a candidate index representing the second block among the candidates included in the merge candidate list may be encoded. If the inter-prediction mode of the first block is AMVP mode, a candidate index representing the second block among the candidates included in the AMVP candidate list may be encoded. If the inter-prediction mode of the first block is affine mode, a candidate index representing the second block among the candidates included in the affine candidate list may be encoded. If the inter-prediction mode of the first block is mmvd mode, a candidate index representing the second block among the candidates included in the candidate list may be encoded.

[0428] Hereinafter, the operation of the video encoding device (3000) performed in relation to the generation and application of the residual model in the embodiments of FIGS. 25 to 27 is described in detail.

[0429] In FIG. 25, the video encoding device (3000) can determine whether to encode the residual between the prediction block and the restoration block of the current block (2510) or to use the residual model of the neighbor block (2520).

[0430] Whether to apply a residual model can be determined by comparing the cost incurred by directly encoding the residuals between the prediction block and the restoration block with the cost of using a residual model. Alternatively, if a residual model is available in a neighboring block (2520), a residual model may be applied to the current block (2510).

[0431] When using a residual model, it can be estimated through a residual model based on the difference between the prediction block and the restoration block of an already encoded neighbor block (2520). The residual model of an already encoded neighbor block (2520) may also be generated based on the difference between the prediction block and the restoration block of the neighbor block (2520).

[0432] At this time, the neighbor block (2520) may be a spatially adjacent block among the candidates that can be included in the candidate list according to the inter-prediction mode of the current block (2510). According to the movement information of the neighbor block (2520), the prediction block of the neighbor block (2520) may also be determined from the reference block (2570) included in the reference picture (2550). A residual model (2530) representing a relationship between the prediction sample of the prediction block of the neighbor block (2520) and the restoration sample of the neighbor block (2520) may be determined. The residual model may be a linear model or a convolutional model.

[0433] When the residual model (2530) of the neighbor block (2520) is applied (2540) to the prediction block of the current block (2510), the recovery samples of the current block (2510) can be determined from the prediction samples of the current block (2510).

[0434] According to one embodiment, the time at which the residual model (2530) of the neighbor block (2520) is applied (2540) to the prediction block of the current block (2510) may be before the encoding of the CTU or slice containing the current block (2510) is completed. In this case, the residual model (2530) is applied to generate recovery samples of the current block (2510), and in-loop filtering may be performed on the recovery samples.

[0435] According to one embodiment, the time at which the residual model (2530) of the neighbor block (2520) is applied (2540) to the prediction block of the current block (2510) may be after the encoding of the CTU or slice containing the current block (2510) is completed. In this case, the prediction block of the current block (2510) becomes the recovery block of the current block (2510) as is, and the residual model (2530) is applied to the recovery block of the current block (2510) as a loop filtering process, thereby generating filtered recovery samples.

[0436] A video encoding device (3000) according to one embodiment may configure and store a residual model between a prediction block and a restoration block when there is a residual between a prediction block and a restoration block of a second block that is in an inter-prediction mode. When the second block is used for inter-prediction of a first block that is encoded later than the second block, the video encoding device (3000) may apply the residual model of the second block to the first block.

[0437] If the residual model is determined based on the difference between the prediction block and the restoration block, for the video encoding device (3000) to use the residual model of the second block after generating the prediction block of the first block, the restoration block of the second block must be determined and the residual model based on the prediction block and the restoration block of the second block must already be configured. If the second block has a residual component, the video encoding device (3000) can generate the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block. On the other hand, if the first block does not have a residual component, the same restoration block as the prediction block of the first block is determined, so the video encoding device (3000) can generate the filtered restoration block of the first block by applying the residual model of the second block to the restoration block of the first block. That is, the residual model of the second block can be applied as a filtering process of the first block.

[0438] When the residual model of the second block is applied as a filtering process of the first block and the first block is referenced for intra prediction of the third block, the restored block before filtering of the first block can be used for intra prediction. Accordingly, the restored block to which the residual model is not applied can be referenced for intra prediction of the third block.

[0439] A video encoding device (3000) according to one embodiment may additionally encode information indicating whether to apply a residual model of a neighboring block to a prediction block of the current block. If information regarding whether to apply a residual model is encoded, the motion information used for inter-prediction of the current block may be limited to motion information including residual information. Therefore, in order to determine the availability of a residual model of a neighboring block, it may be identified whether the neighboring block has available motion information including residual information.

[0440] According to one embodiment, if there is a residual component between a prediction block and a restoration block determined by inter-prediction, the video encoding device (3000) can construct a model for the residual samples between the prediction block and the restoration block. At this time, various relational models can be utilized without being limited to specific models such as linear models or convolutional models.

[0441] A video encoding device (3000) according to one embodiment may indirectly indicate whether there is a residual component between a prediction block and a recovery block in the current block through other information encoded for the current block. For example, whether there is a residual component in the current block may be indicated using a skip mode flag, a merge mode flag, a CBF (coded block flag), etc.

[0442] For example, if the video encoding device (3000) indicates that the encoded CBF of the current block is 1, there is a non-zero residual sample, so the residual-related information of the current block can be encoded. The video encoding device (3000) can generate a residual model based on the difference between the predicted sample and the restored sample of the current block. A relationship of a linear model or a convolutional model for obtaining a restored sample from a predicted sample can be determined.

[0443] A video encoding device (3000) according to one embodiment may determine whether to generate a residual model by using the size of residual samples to reduce the amount of computation required to determine the residual model. For example, even if there are residual components between the prediction samples and the reconstructed samples of the current block, if the sum of all residual samples is not greater than a predetermined threshold, a residual model for the current block may not be generated.

[0444] Information on whether a residual model can be applied to a prediction block can be encoded per block, CTU, slice, and picture.

[0445] For example, information on whether a residual model can be applied to the prediction block of the current block may be encoded and included in a NAL unit corresponding to the coding unit syntax of the current block.

[0446] For example, information on whether a residual model can be applied to the prediction blocks of the blocks currently included in the CTU may be included in the NAL unit corresponding to the coding tree syntax of the CTU.

[0447] For example, information on whether a residual model can be applied to the prediction blocks of the blocks included in the current slice may be included in the NAL unit corresponding to the slice header.

[0448] For example, information on whether a residual model can be applied to the prediction blocks of the blocks included in the current picture may be included in the NAL unit corresponding to the PPS.

[0449] When a residual model is applied, only neighboring blocks possessing a residual model can be available candidate blocks. Therefore, since the candidate list containing neighboring blocks used for inter-prediction includes only available candidate blocks, it can only include neighboring blocks for which motion information containing a residual model has been determined. Neighboring blocks for which a residual model has not been determined cannot be candidate blocks.

[0450] If a surrounding block for which a residual model has not been determined becomes an available candidate block, the video encoding device (3000) does not separately transmit residual information of the surrounding block. The video encoding device (3000) can predict the motion information of the current block and determine the predicted block using only motion information without a residual model of the surrounding block.

[0451] According to one embodiment, when using a surrounding block having motion information including a residual model, the video encoding device (3000) can determine the motion information of the current block using the motion information of the surrounding block. The prediction block can be determined using a reference block pointed to by the motion information of the current block. The video encoding device (3000) may also determine a filtered restoration block by applying the residual model included in the motion information of the surrounding block to the prediction block or restoration block of the current block without the need to additionally encode the residual information of the current block.

[0452] When using a residual model without encoding the residual information of the current block, the residual model may be applied during the filtering step of the current block. According to one embodiment, the residual model may be applied first before loop filtering is applied to the restored block. According to one embodiment, the residual model may be coupled to a specific filtering method, such as an adaptive loop filter (ALF), so that the loop filtering and the application of the residual model may be performed simultaneously.

[0453] A video encoding device (3000) according to one embodiment may encode residual components of a portion of a region within a block and not encode residual components for the remaining region within the block. The video encoding device (3000) may generate a residual model using residual components of a portion of the region, reconstructed samples, and predicted samples. In this case, the residual model may be a linear model or a convolutional model.

[0454] The video encoding device (3000) may not encode residual components for the remaining regions, excluding some regions where residual components are encoded, and may generate only prediction samples for the remaining regions. The video encoding device (3000) may generate reconstructed samples for the remaining regions by applying a residual model generated from some regions to the prediction samples of the remaining regions.

[0455] In FIG. 26, the video encoding device (3000) can encode only the residual components for dark areas (2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 2610, 2611, 2612) in each encoding unit. The video encoding device (3000) can generate a residual model based on the difference between the predicted samples and the restored samples of the dark areas (2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 2610, 2611, 2612). The video encoding device (3000) does not encode the residual components of the bright areas (2651, 2652, 2653, 2654, 2655, 2656, 2657, 2658, 2659, 2660, 2661, 2662). The video encoding device (3000) restores the bright areas (2651, 2652, 2653, 2654, 2655, 2656, 2657, 2658, 2659, 2660, 2661, 2662) by applying a residual model generated from the dark areas (2601, 2602, 2603, 2604, 2605, 2606, 2607, 2608, 2609, 2610, 2611, 2612) to the bright areas (2651, 2652, 2653, 2654, 2655, 2656, 2657, 2658, 2659, 2660, 2661, 2662) Samples can be obtained.

[0456] Additionally, the video encoding device (3000) can encode residual components and generate residual models only in dark regions (2600) spaced apart at predetermined intervals within a block, such as blocks (2620, 2630, 2640, 2670). Residual components for the bright regions (2650) of the blocks (2620, 2630, 2640, 2670) are not encoded. The video encoding device (3000) can determine the restored samples of the bright regions (2650) by applying the residual model of the dark regions (2600) to the predicted samples of the bright regions (2650).

[0457] The residual component can be encoded by performing transform / inverse transform and quantization / inverse quantization on the residual component. If the residual component can be encoded into various forms of regions, such as the blocks exemplified in FIG. 26, information indicating which form of region the residual component was encoded into can be additionally encoded. Alternatively, information regarding the location within the block where the residual component can be encoded can be additionally encoded.

[0458] Accordingly, the video encoding device (3000) can encode residual components only for some regions within the block. The video encoding device (3000) can encode information about regions or locations for encoding residual components.

[0459] The video encoding device (3000) can generate a residual model based on the difference between the predicted samples and the restored samples in some regions where the residual component is encoded.

[0460] The video encoding device (3000) can predict reconstructed samples by applying a residual model to the remaining region where the residual component is not encoded. Reconstructed samples of the remaining region can be predicted by applying a residual model to the predicted samples of the remaining region where the residual component is not encoded.

[0461] FIG. 27 is a diagram illustrating a method for applying a residual model of some CUs within a CTU to the remaining CUs that have no residual components for loop filtering according to one embodiment.

[0462] For the coding units (2712, 2714, 2716, 2718) that are dark regions within the CTU (2700), residual components can be encoded. Therefore, a residual model can be determined using the prediction samples and reconstructed samples of the coding units (2712, 2714, 2716, 2718). On the other hand, for the coding units (2752, 2754, 2756) that are bright regions, residual components are not encoded.

[0463] A video encoding device (3000) according to one embodiment may apply a residual model generated from encoding units (2712, 2714, 2716, 2718) to encoding units (2752, 2754, 2756) which are bright areas. In this case, the residual model may be a linear model or a convolutional model.

[0464] The video encoding device (3000) can generate a residual model for some regions and apply the residual model to the remaining regions within blocks of sizes 256x256, 128x128, 64x64, 32x32, and 16x16, as well as the CTU (2700). Additionally, it can encode information regarding the regions where the residual model is generated, for example, information regarding the location, size, range, etc., of the regions where residual components exist. It can also encode information regarding whether to generate a residual model for each region, and information regarding whether to apply the residual model to the remaining regions. Information regarding whether a residual model is generated in the current region may be information regarding whether a residual model is generated based on the difference between the predicted sample and the reconstructed sample for the current region, or information regarding whether it can be generated. Information regarding whether a residual model is applied to the current region may be information regarding whether a residual model from another region is applied to the predicted samples of the current region, or information regarding whether it can be applied, since the residual component for the current region has not been encoded.

[0465] For example, the video encoding device (3000) selects an area to which a filter is to be applied, such as a picture, frame, slice, CTU, 256x256, 128x128, 64x64, 32x32, 16x16, etc. That is, the area to which a residual model is to be generated or the area to which a residual model is to be applied can be determined for each predetermined data unit.

[0466] At this time, information about the area where the residual model is to be generated or information about the area where the residual model is to be applied may be transmitted from the upper data unit. For example, if the residual model is to be applied per CTU, information about whether it can be applied per CTU is encoded per slice, frame, or picture, and may indicate whether the residual model will be applied to all CTUs included in the slice, frame, or picture.

[0467] In addition, whether a filter is applied to individual region levels can be determined separately. For example, when the CTU is 128x128, information regarding whether a residual model is applied for every 64x64 blocks can be encoded. For example, information regarding whether a residual model is applied for every 64x64 blocks can be sequentially encoded in the front part of the NAL unit corresponding to the CTU. Thus, information regarding whether a residual model is applied for each encoding unit included in the CTU can be encoded so that the video encoding device (3000) can apply the residual model to the encoding unit where the residual model can be applied among the encoding units included in the CTU.

[0468] The video encoding device (3000) can encode information indicating whether to apply a residual model of encoding units included in a picture or slice, and signal it through a picture parameter set or a slice header.

[0469] In addition, the video encoding device (3000) can encode information regarding whether a residual model can be applied to a prediction block for each encoding unit in order to indicate whether a residual model can be applied for each encoding unit.

[0470] For encoding units to which the residual model can be applied, the video encoding device (3000) does not need to separately encode the residual component. Therefore, since the amount of computation and data for decoding the residual component is reduced, more efficient decoding is possible.

[0471] In addition, since information regarding the region where a residual model can be generated can also be encoded on a data unit basis, whether or not to generate a residual model can be determined on a data unit basis.

[0472] A video encoding device (3000) according to one embodiment can generate a residual model at a data unit level where the residual model can be applied. For example, if the residual model can be applied at the CTU level, the residual model can be generated at the CTU level. Therefore, it may be possible to generate a residual model for all encoding units included in the CTU.

[0473] If the application of a residual model is possible at the slice level, the generation of a residual model at the slice level may also be possible. In this case, it may be possible to generate a residual model for all coding units included in the slice.

[0474] If a residual model can be applied at the picture level, it may also be possible to generate the residual model at the picture level. In this case, it may be possible to generate a residual model for all coding units included in the picture.

[0475] As another example, a residual model can be generated at a higher data unit level than the data unit to which the residual model can be applied.

[0476] For example, if a residual model is applicable to each CTU, a residual model can be generated for each slice, frame, or picture containing the CTU. That is, a residual model can be generated from all CTUs included in a slice, frame, or picture.

[0477] For example, if there exists a coding unit among the coding units included in the CTU to which a residual model can be applied, a residual model for the CTU can be generated. If a residual model can be applied to the coding units included in a picture or slice, a residual model can be generated at the picture or slice level.

[0478] In addition, even if information regarding the applicability of the residual model is not separately encoded, whether the residual model is applicable can be determined according to a predetermined rule.

[0479] In addition, even if the application of a residual model to a data unit is possible (applicability), information regarding whether the residual model is applied to that data unit may be encoded separately. Alternatively, even if the application of a residual model to a data unit is possible (applicability), whether the residual model is applied to that data unit may be determined according to a predetermined rule.

[0480] In addition, even if information regarding the possibility of generating a residual model is not separately encoded, whether a residual model can be generated can be determined according to a predetermined rule.

[0481] Based on rules or information regarding the possibility of generating a residual model, whether or not to generate a residual model can be determined if it is possible.

[0482] Additionally, even if the generation of a residual model is possible (applicability) for a data unit according to information or rules regarding whether the generation of a residual model is possible, information regarding whether a residual model is generated for that data unit may be encoded separately. Alternatively, even if the generation of a residual model is possible (generability) for a data unit, whether a residual model is generated for that data unit may be determined according to a predetermined rule.

[0483] A video encoding device (3000) according to one embodiment may encode information regarding whether a residual model can be applied or information regarding whether a residual model is applied, information regarding whether a residual model can be generated or information regarding whether a residual model is generated, respectively. The video encoding device (3000) may encode information for determining a data unit to which a residual model can be applied and / or information regarding whether a residual model can be applied in a current data unit. The video encoding device (3000) may encode information for determining a data unit to which a residual model is applied and / or information regarding whether a residual model is applied in a current data unit. The video encoding device (3000) may encode information for determining a data unit to which a residual model can be generated and / or information indicating whether a residual model can be generated in a current data unit. The video encoding device (3000) may encode information for determining a data unit to which a residual model is generated and / or information regarding whether a residual model is generated in a current data unit.

[0484] A video encoding device (3000) according to one embodiment can generate a residual model for an area where residual information exists among the areas where a residual model can be generated. For example, if the video encoding device (3000) generates prediction samples using two or more reference images, it can generate a final prediction sample using first prediction samples obtained from a first reference image and second prediction samples obtained from a second reference image. In this case, the video encoding device (3000) can generate a residual model using the final prediction samples and the reconstructed samples.

[0485] Additionally, the video encoding device (3000) can apply a residual model to each data unit that is subject to filtering. For example, even if the unit where the residual model is generated is an encoding unit, if the target of filtering is a CTU unit, the residual model can be applied to the CTU unit.

[0486] A video encoding device (3000) according to one embodiment can apply a residual model to areas without residual information among data units to which filtering is applied.

[0487] For example, if the data unit to which filtering is applied is a CTU, a residual model is generated from encoding units within the CTU where residual information exists, and the residual model can be applied to encoding units within the CTU where residual information does not exist.

[0488] As another example, when the data unit to which filtering is applied is a CTU, a residual model can be generated from CTUs where residual information exists, and the residual model can be applied to CTUs where residual information does not exist.

[0489] As another example, the area where the residual model is applied can be the entire data unit to which filtering is applied, regardless of the presence or absence of residual information. Even if residual information exists, if only negligible residuals are present, the additional application of the residual model can be expected to complement the residual components.

[0490] According to one embodiment, a filter to which a residual model is applied can be implemented independently of other filters.

[0491] When a residual model filter is applied in parallel with other filters, the samples output through the residual model's filtering process can become the inputs to the other filters. Additionally, the weighted sum of the outputs of each filter, including the filter to which the residual model is applied and other filters, can be the final output of the filtering.

[0492] FIG. 28 is a diagram illustrating a method for generating a residual model between predicted regions and restored regions within a picture according to one embodiment and applying it to regions without residual information.

[0493] A video encoding device (3000) according to one embodiment may generate and apply a residual model of the entire picture.

[0494] Specifically, the video encoding device (3000) encodes the residual components of the first encoding units (2810) of the dark area within the picture (2800), but does not encode the residual components of the second encoding units (2820) of the bright area.

[0495] The video encoding device (3000) can generate a residual model to be applied to the picture (2800) using the prediction blocks (2840) and the restoration blocks (2850) of the first encoding units (2810). That is, the residual model can also be generated using the first encoding units (2810), which are a plurality of encoding units included in the picture (2800).

[0496] The video encoding device (3000) can obtain filtered samples (2880) of the second encoding units (2820) by applying a residual model generated for the picture (2800) to the second encoding units (2820). Through filtering applied to the prediction samples of the second encoding units (2820) by the residual model, the filtered samples (2880) can be output as restored samples of the second encoding units (2820).

[0497] Similar to the embodiment in FIG. 28 where a residual model is generated and applied at the picture level, the video encoding device (3000) may generate and apply a residual model not only at the encoding unit level but also at the CTU and slice level. Specifically, instead of generating a residual model for each encoding unit included in the CTU, a single residual model may be generated using prediction samples and recovery samples of multiple encoding units that have residual information among the encoding units included in the CTU. In this case, the video encoding device (3000) may obtain filtered recovery samples of each encoding unit that does not have residual information by applying a single residual model to the encoding units that do not have residual information among the encoding units included in the CTU.

[0498] For a residual model to be generated, time is required until the encoding of the reference block is completed. When a residual model is applied block by block, if the encoding completion time of the reference block is later than the generation completion time of the prediction block of the current block, the residual model has not yet been generated. Consequently, a problem may arise where the application of the residual model to the prediction block for the restoration block of the current block is delayed. However, as shown in Fig. 28, when a residual model is applied to a data unit larger than the encoding unit, the target of the filtering to which the residual model is applied expands beyond the encoding unit level to the CTU unit, slice unit, or picture unit. Consequently, the timing of the filtering is also delayed to the encoding completion time of the CTU unit, slice unit, or picture unit. This delay in the timing of the filtering to which the residual model is applied has the advantage of securing time for the residual model to be generated.

[0499] According to one embodiment of the present disclosure, a method for decoding video may include a step of determining that a second block is used for inter-predicting of the first block based on inter-predicting information of the first block obtained from a bitstream. According to one embodiment, a method for decoding video may include a step of identifying whether a residual model of the second block can be used for inter-predicting of the first block. According to one embodiment, a method for decoding video may include a step of generating a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block when the residual model of the second block can be used. According to one embodiment, a method for decoding video may include a step of determining a prediction block of the first block using motion information of the first block, and determining a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

[0500] According to one embodiment, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block may include the step of determining the motion information of the first block using the motion information of the second block. According to one embodiment, the step of determining the restoration block of the first block may include the step of determining the prediction block of the first block using the reference block pointed to by the motion information of the first block. According to one embodiment, the step of determining the restoration block of the first block may include the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block.

[0501] According to one embodiment, the step of determining the restoration block of the first block may include determining the restoration block of the first block using the prediction block of the first block, and generating the filtered restoration block of the first block by applying the residual model of the second block to the restoration block of the first block for in-loop filtering on the restoration block of the first block.

[0502] According to one embodiment, the step of determining the restoration block of the first block may include, after the generation of restoration blocks of blocks included in a coding tree unit (CTU) or slice containing the first block is completed, the step of applying the residual model of the second block to the first block on which it is determined to apply the residual model of the second block.

[0503] According to one embodiment, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block may include the step of determining whether the residual model of the second block can be used by using at least one of a skip mode flag, a merge mode flag, or a CBF (Coded block flag) of the second block. According to one embodiment, the skip mode flag may be information indicating whether the inter-prediction of the second block is performed in skip mode. According to one embodiment, the merge mode flag may be information indicating whether the inter-prediction of the second block is performed in merge mode. According to one embodiment, the CBF may be information indicating whether a non-zero transformation coefficient exists in the second block.

[0504] According to one embodiment, the step of determining whether the residual model of the second block can be used for inter-predicting of the first block may include the step of not generating the residual model of the second block when the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold. According to one embodiment, the step of determining whether the residual model of the second block can be used for inter-predicting of the first block may include the step of generating the residual model of the second block when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold.

[0505] According to one embodiment, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block may include the step of obtaining information from the bitstream indicating whether the residual model of the second block can be used for inter-prediction of the first block for each picture, slice, coding tree unit (CTU), or block containing the first block.

[0506] According to one embodiment of the present disclosure, a method for encoding video may include a step of identifying whether a residual model of a second block can be used for inter-prediction of a first block. According to one embodiment, a method for encoding video may include a step of generating a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block when the residual model of the second block can be used. According to one embodiment, a method for encoding video may include a step of determining a prediction block of the first block and determining a restoration block of the first block by applying the residual model of the second block to the prediction block of the first block. According to one embodiment, a method for encoding video may include a step of encoding information for determining that a surrounding block used for inter-prediction of the first block is the second block.

[0507] In a method for encoding video according to one embodiment, the step of determining a restoration block of the first block by applying a residual model of the second block to a prediction block of the first block may include a step of determining motion information of the first block using motion information of the second block. In a method for encoding video according to one embodiment, the step of determining a restoration block of the first block may include a step of determining a prediction block of the first block using a reference block pointed to by motion information of the first block. In a method for encoding video according to one embodiment, the step of determining a restoration block of the first block may include a step of determining a restoration block of the first block by applying a residual model of the second block to a prediction block of the first block.

[0508] According to one embodiment, in a method for encoding video, the step of determining a restoration block of the first block may include determining a restoration block of the first block using a prediction block of the first block, and generating a filtered restoration block of the first block by applying a residual model of the second block to the restoration block of the first block for in-loop filtering on the restoration block of the first block.

[0509] In a method for encoding video according to one embodiment, the step of determining a recovery block of the first block may include, after the generation of recovery blocks of blocks included in a coding tree unit (CTU) or slice containing the first block is completed, applying the residual model of the second block to the first block on which it is determined to apply the residual model of the second block.

[0510] In a method for encoding video according to one embodiment, the step of identifying whether the residual model of the second block can be used for inter prediction of the first block may include the step of determining whether the residual model of the second block can be used based on whether the inter prediction mode of the second block is a skip mode. In a method for encoding video according to one embodiment, the step of identifying whether the residual model of the second block can be used for inter prediction of the first block may include the step of determining whether the residual model of the second block can be used based on whether the inter prediction mode of the second block is a merge mode. In a method for encoding video according to one embodiment, the step of identifying whether the residual model of the second block can be used for inter prediction of the first block may include the step of determining whether the residual model of the second block can be used based on whether there is a non-zero transform coefficient in the second block.

[0511] According to one embodiment, in a method for encoding video, the step of determining whether the residual model of the second block can be used for inter-prediction of the first block may include the step of not generating the residual model of the second block when the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold. According to one embodiment, in a method for encoding video, the step of determining whether the residual model of the second block can be used for inter-prediction of the first block may include the step of generating the residual model of the second block when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold.

[0512] According to one embodiment, in a method for encoding video, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block may include the step of encoding information indicating whether the residual model of the second block can be used for inter-prediction of the first block for each picture, slice, coding tree unit (CTU), or block containing the first block.

[0513] According to one embodiment of the present disclosure, an apparatus for decoding a video may include a memory for storing instructions and at least one processor operably coupled to the memory and including a processing circuit. An apparatus for decoding a video according to one embodiment in which the at least one processor executes the instructions alone or in cooperation may: determine that a second block is used for inter-predicting of the first block based on inter-predicting information of the first block obtained from the bitstream. An apparatus for decoding a video according to one embodiment may: identify whether a residual model of the second block can be used for inter-predicting of the first block. An apparatus for decoding a video according to one embodiment may: generate a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block when a residual model of the second block can be used. A device for decoding the video according to one embodiment can determine a prediction block of the first block using motion information of the first block, and determine a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

[0514] According to one embodiment of the present disclosure, an apparatus for encoding video may include a memory for storing instructions and at least one processor operably coupled to the memory and comprising a processing circuit. An apparatus for encoding video according to one embodiment, by having the at least one processor execute the instructions alone or in cooperation, may identify whether a residual model of a second block can be used for inter-prediction of a first block. An apparatus for encoding video according to one embodiment may generate a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block when the residual model of the second block can be used. An apparatus for encoding video according to one embodiment may determine a prediction block of the first block and determine a restoration block of the first block by applying the residual model of the second block to the prediction block of the first block. An apparatus for encoding video according to one embodiment may determine that the second block is used for inter-prediction of the first block. An apparatus for encoding the video according to one embodiment can encode information for determining that a surrounding block used for inter-prediction of the first block is the second block.

[0515] According to one embodiment of the present disclosure, a method for transmitting a bitstream may include a step of identifying whether a residual model of a second block can be used for inter-prediction of a first block. According to one embodiment, a method for transmitting a bitstream may include a step of generating a residual model of the second block based on residual information between a prediction block of the second block and a restoration block of the second block when the residual model of the second block can be used. According to one embodiment, a method for transmitting a bitstream may include a step of determining a prediction block of the first block and determining a restoration block of the first block by applying the residual model of the second block to the prediction block of the first block. According to one embodiment, a method for transmitting a bitstream may include a step of outputting a bitstream containing information for determining that a surrounding block used for inter-prediction of the first block is the second block.

[0516] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0517] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Claims

1. Regarding the method of decoding video, A step (2910) of determining that a second block is used for inter-prediction of the first block based on inter-prediction information of the first block obtained from the bitstream; A step (2920) of identifying whether the residual model of the second block can be used for inter-prediction of the first block; When the residual model of the second block is available, a step (2930) of generating the residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block; A method characterized by including the step (2940) of determining a prediction block of the first block using motion information of the first block and determining a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

2. In claim 1, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block is, A step of determining the movement information of the first block using the movement information of the second block; A step of determining a prediction block of the first block using a reference block pointed to by the movement information of the first block; and A method characterized by including the step of determining a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

3. In any one of claims 1 to 2, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block is, A method characterized by including the step of determining a restoration block of the first block using a prediction block of the first block, and generating a filtered restoration block of the first block by applying a residual model of the second block to the restoration block of the first block for in-loop filtering on the restoration block of the first block.

4. In any one of claims 1 to 3, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block is, A method characterized by including the step of applying the residual model of the second block to the first block on which it is decided to apply the residual model of the second block after the generation of restoration blocks of blocks included in a CTU (coding tree unit) or slice including the first block is completed.

5. In any one of claims 1 to 4, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block is, The method includes a step of determining whether the residual model of the second block can be used by utilizing at least one of the skip mode flag, merge mode flag, or CBF (Coded block flag) of the second block. The above skip mode flag is information indicating whether the inter prediction of the second block is performed in skip mode, and The above merge mode flag is information indicating whether the inter prediction of the second block is performed in merge mode, and A method characterized in that the above CBF is information indicating whether a non-zero transformation coefficient exists in the second block.

6. In any one of claims 1 to 5, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block is, A step of not generating a residual model of the second block when the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold; and A method characterized by including the step of generating a residual model of the second block when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold.

7. In any one of claims 1 to 6, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block is, A method characterized by including the step of obtaining from the bitstream information indicating whether a residual model of the second block can be used for inter-prediction of the first block for each picture, slice, CTU (coding tree unit), or block including the first block.

8. In a method for encoding video, A step (3110) of identifying whether the residual model of the second block can be used for inter-prediction of the first block; When the residual model of the second block is available, a step (3120) of generating the residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block; A step (3130) of determining a prediction block of the first block and determining a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block; and A method characterized by including a step (3140) of encoding information for determining that the surrounding block used for inter-prediction of the first block is the second block.

9. In claim 8, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block is, A step of determining the movement information of the first block using the movement information of the second block; A step of determining a prediction block of the first block using a reference block pointed to by the movement information of the first block; and A method characterized by including the step of determining a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block.

10. In any one of claims 8 to 9, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block, A method characterized by including the step of determining a restoration block of the first block using a prediction block of the first block, and generating a filtered restoration block of the first block by applying a residual model of the second block to the restoration block of the first block for in-loop filtering on the restoration block of the first block.

11. In any one of claims 8 to 10, the step of determining the restoration block of the first block by applying the residual model of the second block to the prediction block of the first block, A method characterized by including the step of applying the residual model of the second block to the first block on which it is decided to apply the residual model of the second block after the generation of restoration blocks of blocks included in a CTU (coding tree unit) or slice including the first block is completed.

12. In any one of claims 8 to 11, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block is, A step of determining whether the residual model of the second block can be used based on whether the inter-prediction mode of the second block is a skip mode; A step of determining whether the residual model of the second block can be used based on whether the inter-prediction mode of the second block is a merge mode; A method characterized by including a step of determining whether a residual model of the second block can be used based on whether a non-zero transformation coefficient exists in the second block.

13. In any one of claims 8 to 12, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block is, A step of not generating a residual model of the second block when the sum of the values ​​of the residual samples of the second block is less than or equal to a predetermined threshold; and A method characterized by including the step of generating a residual model of the second block when the sum of the values ​​of the residual samples of the second block is greater than a predetermined threshold.

14. In any one of claims 8 to 13, the step of identifying whether the residual model of the second block can be used for inter-prediction of the first block is, A method characterized by including the step of encoding information indicating whether a residual model of the second block can be used for inter-prediction of the first block for each picture, slice, CTU (coding tree unit), or block including the first block.

15. Regarding the method of transmitting a bitstream, A step of identifying whether the residual model of the second block can be used for inter-prediction of the first block; When the residual model of the second block is available, a step of generating the residual model of the second block based on residual information between the prediction block of the second block and the restoration block of the second block; A step of determining a prediction block of the first block and determining a restoration block of the first block by applying a residual model of the second block to the prediction block of the first block; and A method characterized by including the step of outputting a bitstream containing information for determining that a surrounding block used for inter-predicting of the first block is the second block.

Images