Image encoding / decoding method and device, and method for transmitting bitstream

Abstract

An image encoding / decoding method and device, and a method for transmitting a bitstream are provided. The image decoding method comprises the steps of: acquiring, from a bitstream, one or more syntax elements for determining a residual coding syntax structure; determining any one from among a plurality of residual coding syntax structures as a residual coding syntax structure for the current block on the basis of the one or more syntax elements; and acquiring a residual coefficient of the current block on the basis of the determined residual coding syntax structure, wherein the one or more syntax elements can include a syntax element indicating whether the current block is predicted on the basis of neural network-based intra prediction (NNIP).

Description

Video encoding / decoding method, device, and method for transmitting a bitstream

[0001] The present disclosure relates to an image encoding / decoding method, an apparatus, and a method for transmitting a bitstream. Specifically, the present disclosure relates to an image encoding / decoding method, an apparatus, and a method for transmitting a bitstream based on a method for determining residual coding and entropy coding schemes.

[0002] Recently, the demand for high-resolution, high-quality video, such as UHD (Ultra High Definition) video, is increasing across various application fields. Furthermore, interest in and demand for immersive media, including VR (Virtual Reality), AR (Artificial Reality), and holograms, are also on the rise. Additionally, the broadcasting of video with characteristics distinct from reality, such as game footage, is also increasing. As video data becomes higher in resolution and quality, the relative volume of data increases compared to conventional video data; consequently, transmission and storage costs rise when video data is transmitted using existing wired or wireless broadband lines or stored using conventional storage media. To address these issues arising from the increase in data resolution and quality, high-efficiency video encoding and decoding technologies for video with higher resolution and quality are required.

[0003] The statistical characteristics of residual signals are becoming increasingly complex due to the introduction of various prediction modes and transformation methods. Consequently, problems such as increased complexity, reduced coding efficiency, or processing delays may arise during the residual coding and entropy coding processes.

[0004] The present disclosure aims to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] In addition, the present disclosure aims to provide a recording medium storing a bitstream generated by an image decoding method or device according to the present disclosure, and a method for transmitting the bitstream.

[0006] In addition, the present disclosure aims to provide an encoding and decoding method that can more flexibly apply a residual coding structure or an entropy coding method depending on the characteristics of the residual signal or the encoding environment in order to solve the above-mentioned problems.

[0007] A video decoding method according to one embodiment of the present disclosure comprises the steps of: obtaining one or more syntax elements for determining a residual coding syntax structure from a bitstream; determining one of a plurality of residual coding syntax structures as a residual coding syntax structure for a current block based on the one or more syntax elements; and obtaining residual coefficients of the current block based on the determined residual coding syntax structure, wherein the one or more syntax elements may include a syntax element indicating whether the current block was predicted based on neural network-based intra-prediction (NNIP).

[0008] In the above image decoding method, the one or more syntax elements can be obtained from at least one of a picture header (PH) and a slice header (SH).

[0009] In the above image decoding method, the plurality of residual coding syntax structures may differ from each other in at least one of the set of syntax elements used to obtain the residual coefficient of the current block, whether the syntax elements are conditionally included, and the signaling order of the syntax elements.

[0010] In the above-described image decoding method, the one or more syntax elements may include a syntax element indicating whether the current block was predicted based on an in-frame prediction or a cross-frame prediction.

[0011] In the above image decoding method, the one or more syntax elements may include a syntax element indicating at least one of the width and height of the current block.

[0012] In the above image decoding method, the one or more syntax elements include a syntax element indicating a transformation method for the current block, and the syntax element indicating the transformation method may include a first transformation syntax element indicating whether to perform a secondary transform and a second transformation syntax element indicating whether the primary transform is a separable transform or a non-separable transform.

[0013] In the above image decoding method, the determined residual coding syntax structure can be used when obtaining residual coefficients for a plurality of blocks included in the same slice or the same picture as the current block.

[0014] In the above image decoding method, the determining step may include the step of determining one of the plurality of residual coding syntax structures as a first residual coding syntax structure for the luminance (luma) component of the current block and the step of determining one of the plurality of residual coding syntax structures as a second residual coding syntax structure for the chroma component of the current block.

[0015] In the above image decoding method, the step of obtaining the residual coefficient may include the step of obtaining the residual coefficient of the luminance component based on the first residual coding syntax structure and the step of obtaining the residual coefficient of the chrominance component based on the second residual coding syntax structure.

[0016] In the above image decoding method, the syntax element can be obtained at a level corresponding to at least one of a Coding tree unit (CTU), a Coding unit (CU), a Prediction unit (PU), or a Transform unit (TU).

[0017] A video encoding method according to one embodiment of the present disclosure comprises the steps of determining one or more syntax elements for determining a residual coding syntax structure, determining one of a plurality of residual coding syntax structures as a residual coding syntax structure for a current block based on the one or more syntax elements, and encoding residual coefficients of the current block based on the determined residual coding syntax structure, wherein the one or more syntax elements may include a syntax element indicating whether the current block was predicted based on neural network-based intra-prediction (NNIP).

[0018] A non-transient computer-readable recording medium storing a bitstream generated by an image encoding method according to one embodiment of the present disclosure can store the bitstream generated by the image encoding method.

[0019] A bitstream transmission method according to one embodiment of the present disclosure can transmit a bitstream generated by the image encoding method.

[0020] The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

[0021] According to the present disclosure, an image encoding / decoding method and apparatus with improved encoding / decoding efficiency may be provided.

[0022] In addition, according to the present disclosure, the complexity in the residual coding and entropy coding processes can be reduced and the coding efficiency can be increased.

[0023] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0024] FIG. 1 is a block diagram showing the configuration according to one embodiment of an encoding device to which the present disclosure applies.

[0025] FIG. 2 is a block diagram showing the configuration according to one embodiment of a decoding device to which the present disclosure is applied.

[0026] FIG. 3 is a schematic diagram illustrating a video coding system to which the present disclosure can be applied.

[0027] Figures 4 to 7 are drawings for explaining examples of residual coding structures.

[0028] FIG. 8 is a flowchart illustrating a decoding method according to one embodiment of the present disclosure.

[0029] FIG. 9 is a flowchart illustrating an encoding method according to one embodiment of the present disclosure.

[0030] FIG. 10 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0031] A video decoding method according to one embodiment of the present disclosure comprises the steps of: obtaining one or more syntax elements for determining a residual coding syntax structure from a bitstream; determining one of a plurality of residual coding syntax structures as a residual coding syntax structure for a current block based on the one or more syntax elements; and obtaining residual coefficients of the current block based on the determined residual coding syntax structure, wherein the one or more syntax elements may include a syntax element indicating whether the current block was predicted based on neural network-based intra-prediction (NNIP).

[0032] The present disclosure is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be provided illustratively for clearer explanation. The detailed description of the exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0033] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0034] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not deviate from the essence of the present disclosure.

[0035] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, some components of this disclosure may not be essential components performing an essential function in this disclosure, but may be optional components merely for enhancing performance. This disclosure may be implemented by including only the components essential to embody the essence of this disclosure, excluding components used merely for performance enhancement, and a structure including only the essential components, excluding optional components used merely for performance enhancement, is also included within the scope of this disclosure.

[0036] In the embodiments, the term "at least one" may mean one of a number of 1 or more, such as 1, 2, 3, and 4. In the embodiments, the term "a plurality of" may mean one of a number of 2 or more, such as 2, 3, and 4.

[0037] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0038] Glossary of Terms

[0039] In the following, “image” may refer to a single picture constituting a video, or it may refer to the video itself. For example, “encoding and / or decoding of an image” may mean “encoding and / or decoding of an image”, and may also mean “encoding and / or decoding of one of the images constituting the video”.

[0040] In the following, "video" and "video" may be used interchangeably with the same meaning. Additionally, the target image may be an image to be encoded and / or an image to be decoded. Furthermore, the target image may be an input image fed into an encoding device and an input image fed into a decoding device. Here, the target image may have the same meaning as the current image.

[0041] In the following, the terms encoder and image encoding device may be used interchangeably.

[0042] In the following, the decoder and the image decoder may be used interchangeably with each other.

[0043] In the following, "image," "picture," "frame," and "screen" may be used interchangeably with the same meaning.

[0044] In the following, “target block” may be an encoding target block that is the target of encoding and / or a decoding target block that is the target of decoding. Additionally, the target block may be a current block that is the target of current encoding and / or decoding. For example, “target block” and “current block” may be used interchangeably.

[0045] In the following description, "block" and "unit" may be used interchangeably. Additionally, to distinguish it from a block, "unit" may refer to a block containing a luminance (Luma) component block and a corresponding chroma (Chroma) component block. For example, a Coding Tree Unit (CTU) may consist of a single luminance component (Y) coding tree block (CTB) and two chroma component (Cb, Cr) coding tree blocks associated with it.

[0046] In the following, “sample,” “pixel,” and “pixel” may be used interchangeably with the same meaning. Here, a sample may represent a basic unit constituting a block.

[0047] In the following, “inter” and “inter-screen” may be used interchangeably with the same meaning.

[0048] In the following, “intra” and “in-screen” may be used interchangeably with the same meaning.

[0049]

[0050] FIG. 1 is a block diagram showing the configuration according to one embodiment of an encoding device to which the present disclosure applies.

[0051] The encoding device (100) may be an encoder, a video encoding device, or an image encoding device. The video may include one or more images. The encoding device (100) may sequentially encode one or more images.

[0052] Referring to FIG. 1, the encoding device (100) may include an image segmentation unit (110), an intra prediction unit (120), a motion prediction unit (121), a motion compensation unit (122), a switch (115), a subtractor (113), a converter (130), a quantization unit (140), an entropy encoding unit (150), an inverse quantization unit (160), an inverse converter (170), an adder (117), a filter unit (180), and a reference picture buffer (190).

[0053] Additionally, the encoding device (100) can generate a bitstream containing encoded information through encoding of an input image and can output the generated bitstream. The generated bitstream can be stored on a computer-readable recording medium or streamed via a wired / wireless transmission medium.

[0054] The video segmentation unit (110) can divide the input video into various forms to increase the efficiency of video encoding / decoding. That is, the input video consists of multiple pictures, and a single picture can be processed by hierarchically dividing it for compression efficiency, parallel processing, etc. For example, a single picture can be divided into one or more tiles or slices and then divided again into multiple CTUs (Coding Tree Units). Alternatively, a single picture can first be divided into multiple sub-pictures defined as groups of rectangular slices, and each sub-picture can be divided into the said tiles / slices. Here, the sub-pictures can be utilized to support the function of partially and independently encoding / decoding and transmitting the picture. Since multiple sub-pictures can each be restored individually, they have the advantage of being easy to edit in applications that configure multi-channel inputs into a single picture. In addition, the tiles can be divided horizontally to create bricks. Here, a brick can be utilized as the basic unit of parallel processing within a picture. Additionally, a single CTU can be recursively partitioned into a Quadtree (QT), and the terminal node of the partition can be defined as a Coding Unit (CU). The CU can be divided into a Prediction Unit (PU) and a Transform Unit (TU) to perform prediction and partitioning. Meanwhile, the CU can be utilized as the prediction unit and / or the transformation unit itself. Here, for flexible partitioning, each CTU can be recursively partitioned into a Multi-Type Tree (MTT) as well as a Quadtree (QT). The partitioning of the CTU into a Multi-Type Tree can begin at the terminal node of the QT, and the MTT can be composed of a Binary Tree (BT) and a Triple Tree (TT).For example, the MTT structure can be divided into vertical binary splitting mode (SPLIT_BT_VER), horizontal binary splitting mode (SPLIT_BT_HOR), vertical ternary splitting mode (SPLIT_TT_VER), and horizontal ternary splitting mode (SPLIT_TT_HOR). Additionally, when splitting, the minimum block size (MinQTSize) of the quad tree for the luminance block can be set to 16x16, the maximum block size (MaxBtSize) of the binary tree to 128x128, and the maximum block size (MaxTtSize) of the triple tree to 64x64. Furthermore, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the triple tree can be set to 4x4, and the maximum depth (MaxMttDepth) of the multitype tree can be set to 4. Additionally, to increase the encoding efficiency of the I slice, a dual tree can be applied that uses different CTU splitting structures for the luminance and chrominance components. On the other hand, in P and B slices, the luminance and color difference CTBs (Coding Tree Blocks) within the CTU can be divided into a single tree that shares a coding tree structure.

[0055] The encoding device (100) may perform encoding on an input image in an intra mode and / or inter mode. Alternatively, the encoding device (100) may perform encoding on an input image in a third mode other than the intra mode and inter mode (e.g., IBC mode, Palette mode, etc.). However, if the third mode has functional characteristics similar to the intra mode or inter mode, it may be classified as an intra mode or inter mode for convenience of explanation. In this disclosure, the third mode will be classified and described separately only when a specific description of the third mode is required.

[0056] When intra mode is used as the prediction mode, the switch (115) can be switched to intra, and when inter mode is used as the prediction mode, the switch (115) can be switched to inter. Here, intra mode may mean an intra-frame prediction mode, and inter mode may mean an inter-frame prediction mode. The encoding device (100) can generate a prediction block for an input block of an input image. Additionally, after the prediction block is generated, the encoding device (100) can encode a residual block using the residual of the input block and the prediction block. The input image may be referred to as the current image that is the subject of current encoding. The input block may be referred to as the current block that is the subject of current encoding or the encoding target block.

[0057] When the prediction mode is an intra mode, the intra prediction unit (120) may use a sample of a block that has already been encoded / decoded around the current block as a reference sample. The intra prediction unit (120) may perform spatial prediction for the current block using the reference sample and generate prediction samples for the input block through spatial prediction. Here, intra prediction may mean intra-frame prediction.

[0058] In the intra prediction method, non-directional prediction modes such as DC mode and Planar mode, and directional prediction modes (e.g., 65 directions) may be applied. Here, the intra prediction method can be expressed as an intra prediction mode or an intra-frame prediction mode.

[0059] When the prediction mode is an inter mode, the motion prediction unit (121) can search for the region that best matches the input block from the reference image during the motion prediction process and derive a motion vector using the searched region. At this time, the search region can be used as the region. The reference image can be stored in the reference picture buffer (190). Here, the reference image can be stored in the reference picture buffer (190) when encoding / decoding of the reference image is processed.

[0060] The motion compensation unit (122) can generate a prediction block for the current block by performing motion compensation using a motion vector. Here, inter-prediction may mean inter-frame prediction or motion compensation.

[0061] The motion prediction unit (121) and motion compensation unit (122) can generate a prediction block by applying an interpolation filter to a portion of the reference image when the value of the motion vector does not have an integer value. To perform inter-frame prediction or motion compensation, based on the encoding unit, it can determine whether the motion prediction and motion compensation method of the prediction unit included in the corresponding encoding unit is a Skip Mode, Merge Mode, Advanced Motion Vector Prediction (AMVP) Mode, or Intra Block Copy (IBC) Mode, and can perform inter-frame prediction or motion compensation according to each mode.

[0062] In addition, based on the above-mentioned inter-frame prediction method, the AFFINE mode of sub-PU-based prediction, the SbTMVP (Subblock-based Temporal Motion Vector Prediction) mode, and the MMVD (Merge with MVD) mode and GPM (Geometric Partitioning Mode) mode of PU-based prediction may be applied. Furthermore, to improve the performance of each mode, HMVP (History based MVP), PAMVP (Pairwise Average MVP), CIIP (Combined Intra / Inter Prediction), AMVR (Adaptive Motion Vector Resolution), BDOF (Bi-Directional Optical-Flow), BCW (Bi-predictive with CU Weights), LIC (Local Illumination Compensation), TM (Template Matching), OBMC (Overlapped Block Motion Compensation), etc. may be applied.

[0063] Among these, AFFINE mode is a technology used in both AMVP and MERGE modes and also offers high encoding efficiency. Conventional video coding standards have the disadvantage of failing to properly compensate for real-world movements, such as zoom in / out and rotation, because they perform Motion Compensation (MC) by considering only the translation of blocks. To address this, a 4-parameter affine motion model using two control point motion vectors (CPMV) and a 6-parameter affine motion model using three control point motion vectors can be applied to inter-prediction. Here, CPMV is a vector representing one of the affine motion models of the top-left, top-right, or bottom-left corners of the current block.

[0064] The subtractor (113) can generate a residual block using the difference between the input block and the prediction block. The residual block may also be referred to as a residual signal. The residual signal may represent the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming, quantizing, or both transforming and quantizing the difference between the original signal and the prediction signal. The residual block may be a residual signal in block units.

[0065] The transformation unit (130) can generate a transform coefficient by performing a transform on the remaining block and output the generated transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transform on the remaining block. When a transform skip mode is applied, the transformation unit (130) may skip the transform on the remaining block.

[0066] A quantized level can be generated by applying quantization to a conversion coefficient or a residual signal. In the following embodiments, the quantized level may also be referred to as a conversion coefficient.

[0067] For example, a 4x4 luminance residual block generated through intra prediction can be transformed using a Discrete Sine Transform (DST)-based basis vector, while the remaining residual blocks can be transformed using a Discrete Cosine Transform (DCT)-based basis vector. Additionally, the transformation blocks for a single block can be divided into a quad tree form using Residual Quad Tree (RQT) technology, and after performing transformation and quantization on each transformation block divided by RQT, a coded block flag (cbf) can be transmitted to increase coding efficiency in the case where all coefficients become zero.

[0068] As another alternative, the Multiple Transform Selection (MTS) technique can be applied to perform transformations using multiple transformation bases selectively. In other words, instead of dividing a CU into TUs via RQT, a function similar to TU division can be performed using the Sub-block Transform (SBT) technique. Specifically, SBT is applied only to inter-frame prediction blocks and, unlike RQT, divides the current block into ½ or ¼ sizes in the vertical or horizontal direction, and then performs a transformation on only one of the blocks. For example, if divided vertically, a transformation can be performed on the leftmost or rightmost block, and if divided horizontally, a transformation can be performed on the topmost or bottommost block.

[0069] In addition, Low Frequency Non-Separable Transform (LFNST), a secondary transform technique that further transforms the residual signal converted to the frequency domain through DCT or DST, can also be applied. LFNST performs additional transformation on the 4x4 or 8x8 low-frequency region in the upper left corner, thereby allowing the residual coefficients to be concentrated in the upper left corner.

[0070] The quantization unit (140) can generate a quantized level by quantizing a transformation coefficient or residual signal according to a quantization parameter (QP, Quantization parameter) and can output the generated quantized level. At this time, the quantization unit (140) can quantize the transformation coefficient using a quantization matrix.

[0071] For example, a quantizer using QP values ​​from 0 to 51 can be used. Alternatively, if the image size is larger and higher coding efficiency is required, QP values ​​from 0 to 63 can be used. Additionally, a Dependent Quantization (DQ) method using two quantizers instead of a single one can be applied. DQ performs quantization using two quantizers (e.g., Q0, Q1), but can be applied so that the quantizer to be used for the next transform coefficient is selected based on the current state through a state transition model, even without signaling information regarding the use of a specific quantizer.

[0072] The entropy encoding unit (150) can generate a bitstream and output a bitstream by performing entropy encoding according to a probability distribution on values ​​calculated by the quantization unit (140) or coding parameter values ​​calculated during the encoding process. The entropy encoding unit (150) can perform entropy encoding on information regarding a sample of an image and information for decoding an image. For example, information for decoding an image may include syntax elements, etc.

[0073] When entropy coding is applied, a small number of bits are allocated to symbols with a high probability of occurrence and a large number of bits are allocated to symbols with a low probability of occurrence, thereby representing the symbols and reducing the size of the bit sequence for the symbols to be encoded. The entropy coding unit (150) may use encoding methods such as exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) for entropy coding. For example, the entropy coding unit (150) may perform entropy coding using a Variable Length Coding (VLC) table. In addition, the entropy encoding unit (150) may perform arithmetic encoding using the derived binarization method, probability model, and context model after deriving a binarization method of the target symbol and a probability model of the target symbol / bin.

[0074] In this regard, when applying CABAC, in order to reduce the size of the probability table stored in the decoder, the table probability update method may be changed to a table update method using a simple formula. In addition, two different probability models may be used to obtain more accurate symbol probability values.

[0075] The entropy encoding unit (150) can convert a 2-dimensional block form coefficient into a 1-dimensional vector form through a transform coefficient scanning method to encode a transform coefficient level (quantized level).

[0076] Coding parameters may include information (flags, indexes, etc.) that is encoded in the encoding device (100) and signaled to the decoding device (200), such as syntax elements, as well as information derived during the encoding process or decoding process, and may refer to information required when encoding or decoding images.

[0077] Here, signaling a flag or index may mean that in an encoder, the corresponding flag or index is entropy encoded and included in a bitstream, and in a decoder, the corresponding flag or index is entropy decoded from the bitstream.

[0078] The encoded current image can be used as a reference image for other images processed later. Accordingly, the encoding device (100) can restore or decode the encoded current image again, and can store the restored or decoded image as a reference image in the reference picture buffer (190).

[0079] The quantized level can be dequantized in the dequantization unit (160) and inverse transformed in the inverse transform unit (170). The dequantized and / or inverse transformed coefficients can be added to the prediction block through the adder (117). A reconstructed block can be generated by adding the dequantized and / or inverse transformed coefficients and the prediction block. Here, the dequantized and / or inverse transformed coefficients refer to coefficients for which at least one of dequantization and inverse transformation has been performed, and may refer to the reconstructed residual block. The dequantization unit (160) and the inverse transform unit (170) can be performed as the reverse process of the quantization unit (140) and the transformation unit (130).

[0080] The restoration block may pass through a filter section (180). The filter section (180) may apply a deblocking filter, Sample Adaptive Offset (SAO), Adaptive Loop Filter (ALF), Bilateral filter (BIF), LMCS (Luma Mapping with Chroma Scaling), etc., to the restoration sample, restoration block, or restoration image as a whole or part of the filtering technique. The filter section (180) may also be referred to as an in-loop filter. In this case, the term in-loop filter is also used as a name that excludes LMCS.

[0081] Deblocking filters can remove block distortion occurring at the boundaries between blocks. To determine whether to perform deblocking, the decision to apply the filter to the current block can be made based on samples contained in a few columns or rows within the block. When applying a deblocking filter to a block, different filters can be applied depending on the required deblocking filtering intensity.

[0082] To compensate for encoding errors using a sample adaptive offset, an appropriate offset value can be added to the sample value. The sample adaptive offset can correct the offset from the original image on a sample-by-sample basis for the deblocked image. One method may be to divide the samples included in the image into a certain number of regions, determine the region to be offset, and apply the offset to that region, or to apply the offset by considering the edge information of each sample.

[0083] A bilateral filter (BIF) can also correct the offset from the original image on a sample-by-sample basis for the deblocked image.

[0084] An adaptive loop filter can perform filtering based on a comparison of the reconstructed image and the original image. After dividing the samples included in the image into predetermined groups, a filter to be applied to each group can be determined, thereby performing filtering differently for each group. Information regarding whether to apply an adaptive loop filter can be signaled per coding unit (CU), and the shape and filter coefficients of the adaptive loop filter to be applied may vary depending on each block.

[0085] In LMCS (Luma Mapping with Chroma Scaling), Luma mapping (LM) refers to remapping luminance values ​​through a piece-wise linear model, and Chroma scaling (CS) refers to a technique that scales the residual values ​​of the chrominance component according to the average luminance value of the predicted signal. In particular, LMCS can be utilized as an HDR correction technique that reflects the characteristics of HDR (High Dynamic Range) video.

[0086] The restored block or restored image that has passed through the filter unit (180) can be stored in the reference picture buffer (190). The restored block that has passed through the filter unit (180) may be part of the reference image. That is to say, the reference image may be a restored image composed of the restored blocks that have passed through the filter unit (180). The stored reference image may subsequently be used for inter-frame prediction or motion compensation.

[0087] FIG. 2 is a block diagram showing the configuration according to one embodiment of a decoding device to which the present disclosure is applied.

[0088] The decoding device (200) may be a decoder, a video decoding device, or an image decoding device.

[0089] Referring to FIG. 2, the decoding device (200) may include an entropy decoding unit (210), an inverse quantization unit (220), an inverse transformation unit (230), an intra prediction unit (240), a motion compensation unit (250), an adder (201), a switch (203), a filter unit (260), and a reference picture buffer (270).

[0090] The decoding device (200) can receive a bitstream output from the encoding device (100). The decoding device (200) can receive a bitstream stored in a computer-readable recording medium or a bitstream stream streamed through a wired / wireless transmission medium. The decoding device (200) can perform decoding on the bitstream in intra mode or inter mode. Additionally, the decoding device (200) can generate a restored image or a decoded image through decoding and can output the restored image or the decoded image.

[0091] If the prediction mode used for decoding is intra mode, the switch (203) can be switched to intra. If the prediction mode used for decoding is inter mode, the switch (203) can be switched to inter.

[0092] The decoding device (200) can decode the input bitstream to obtain a reconstructed residual block and generate a prediction block. Once the reconstructed residual block and the prediction block are obtained, the decoding device (200) can generate a reconstructed block to be decoded by adding the reconstructed residual block and the prediction block. The block to be decoded may be referred to as the current block.

[0093] The entropy decoding unit (210) can generate symbols by performing entropy decoding according to the probability distribution of the bitstream. The generated symbols may include symbols in the form of quantized levels. Here, the entropy decoding method may be the inverse process of the entropy encoding method described above.

[0094] The entropy decoding unit (210) can convert a one-dimensional vector-shaped coefficient into a two-dimensional block-shaped coefficient through a conversion coefficient scanning method to decode a conversion coefficient level (quantized level).

[0095] The quantized level can be dequantized in the dequantization unit (220) and inversely transformed in the inverse transformation unit (230). The quantized level can be generated as a restored residual block as a result of performing dequantization and / or inverse transformation. At this time, the dequantization unit (220) can apply a quantization matrix to the quantized level. The dequantization unit (220) and the inverse transformation unit (230) applied to the decoding device can apply the same technology as the dequantization unit (160) and the inverse transformation unit (170) applied to the aforementioned encoding device.

[0096] When an intra mode is used, the intra prediction unit (240) can generate a prediction block by performing a spatial prediction on the current block using sample values ​​of already decoded blocks around the block to be decoded. The intra prediction unit (240) applied to the decoding device can apply the same technology as the intra prediction unit (120) applied to the aforementioned encoding device.

[0097] When an inter mode is used, the motion compensation unit (250) can generate a prediction block by performing motion compensation on the current block using a motion vector and a reference image stored in the reference picture buffer (270). The motion compensation unit (250) can generate a prediction block by applying an interpolation filter to a portion of the reference image when the value of the motion vector does not have an integer value. To perform motion compensation, it can be determined whether the motion compensation method of the prediction unit included in the corresponding encoding unit is a skip mode, merge mode, AMVP mode, or current picture reference mode based on the encoding unit, and motion compensation can be performed according to each mode. The motion compensation unit (250) applied to the decoder can apply the same technology as the motion compensation unit (122) applied to the aforementioned encoding unit.

[0098] The adder (201) can generate a restored block by adding the restored residual block and the prediction block. The filter unit (260) can apply at least one of the following to the restored block or the restored image: an inverse-LMCS, a deblocking filter, a sample adaptive offset, and an adaptive loop filter. The filter unit (260) applied to the decoder can apply the same filtering technology as the filter unit (180) applied to the aforementioned encoding device.

[0099] The filter unit (260) can output a restored image. The restored block or the restored image can be stored in a reference picture buffer (270) and used for inter-frame prediction. The restored block that has passed through the filter unit (260) may be part of the reference image. That is to say, the reference image may be a restored image composed of the restored blocks that have passed through the filter unit (260). The stored reference image may subsequently be used for inter-frame prediction or motion compensation.

[0100] FIG. 3 is a schematic diagram illustrating a video coding system to which the present disclosure can be applied.

[0101] A video coding system according to one embodiment may include an encoding device (10) and a decoding device (20). The encoding device (10) may transmit encoded video and / or image information or data to the decoding device (20) via a digital storage medium or network in the form of a file or streaming.

[0102] An encoding device (10) according to one embodiment may include a video source generation unit (11), an encoding unit (12), and a transmission unit (13). A decoding device (20) according to one embodiment may include a receiving unit (21), a decoding unit (22), and a rendering unit (23). The encoding unit (12) may be called a video / image encoding unit, and the decoding unit (22) may be called a video / image decoding unit. The transmission unit (13) may be included in the encoding unit (12). The receiving unit (21) may be included in the decoding unit (22). The rendering unit (23) may include a display unit, and the display unit may be composed of a separate device or an external component.

[0103] The video source generation unit (11) can acquire video / image through a process of capturing, synthesizing, or generating video / image. The video source generation unit (11) may include a video / image capture device and / or a video / image generation device. The video / image capture device may include, for example, one or more cameras, a video / image archive containing previously captured video / image, etc. The video / image generation device may include, for example, a computer, a tablet, and a smartphone, etc., and can generate video / image (electronically). For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process of generating related data.

[0104] The encoding unit (12) can encode the input video / image. The encoding unit (12) can perform a series of procedures such as prediction, conversion, and quantization for compression and encoding efficiency. The encoding unit (12) can output the encoded data (encoded video / image information) in the form of a bitstream. The detailed configuration of the encoding unit (12) can also be configured in the same way as the encoding device (100) of FIG. 1 described above.

[0105] The transmission unit (13) can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit (21) of the decoding device (20) via a digital storage medium or network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (13) may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit (21) can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit (22).

[0106] The decoding unit (22) can decode a video / image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operation of the encoding unit (12). The detailed configuration of the decoding unit (22) can also be configured to be identical to the decoding device (200) of FIG. 2 described above.

[0107] The rendering unit (23) can render the decoded video / image. The rendered video / image can be displayed through the display unit.

[0108]

[0109] Residual coding may refer to a set of syntax elements and a coding method for efficiently compressing residuals within any block, region, or unit. In the case of lossy compression, residuals may be expressed as transform coefficients or transform skip coefficients depending on the transform and / or transform skip. In the case of lossless compression, the residuals may be defined as the difference between the original values ​​and predicted values ​​of the image.

[0110] In this specification, a residual coding structure or a residual-coding syntax structure is a syntax structure for encoding and / or decoding residual coefficients, and may mean a concept comprising a set of one or more syntax elements encoded / decoded in relation to the residual, conditional inclusion and signaling level of each syntax element, signaling order and hierarchical arrangement of syntax elements, dependency relationships and derivation rules between syntax elements.

[0111] FIGS. 4 to 7 are drawings for illustrating examples of residual coding structures. A coding structure according to the present specification may correspond to a syntactic structure that is identical or similar to the residual coding structure of FIGS. 4 to 7. However, the residual coding structure is not limited to a specific standard and may include various variations of syntactic elements and their syntactic arrangements / rules defined for encoding and / or decoding residual coefficients.

[0112]

[0113] In this specification, "coding of syntax element" may refer to a series of processes in which, for a syntax element defined according to a residual coding structure, the value or state of the corresponding syntax element is determined, and the determined value is entropy-coded so as to be included in a bitstream in an encoding device, or the value of the syntax element is obtained or derived based on information included in the bitstream in a decoding device.

[0114] For example, the coding of a syntax element may include a flag indicating the existence of the syntax element, an index indicating the value or range of the syntax element, or a numerical value, and said value may be encoded or decoded according to an entropy coding method. Additionally, the coding of a syntax element may include not only cases where the value of the syntax element is explicitly signaled in the bitstream, but also cases where it is derived by a decoder based on other syntax elements, parameters, or peripheral information contained in the bitstream.

[0115] Entropy coding is a representative form of lossless data compression and may refer to an encoding and / or decoding process in which each symbol or sequence of symbols is represented as a codeword of different lengths according to the probability of occurrence of the symbol.

[0116] For example, context-based adaptive variable length coding based on variable length code, context-based adaptive binary arithmetic coding based on arithmetic coding, and probability interval partitioning entropy (PIPE) can be applied as examples of entropy coding.

[0117] In this case, entropy encoding syntax elements or encoding parameters in the encoding device and including them in the bitstream may refer to entropy encoding, and entropy decoding syntax elements or encoding parameters from the bitstream in the decoding device may refer to entropy decoding.

[0118] In recent video encoding and decoding technologies, the statistical characteristics of residual signals are becoming increasingly diverse due to the introduction of various prediction modes, transform methods, quantization methods, and additional image processing tools. Consequently, if a fixed residual coding structure or a single entropy coding method is uniformly applied as in the past, limitations in coding efficiency may arise, or increased complexity or processing delays may occur during the encoding and / or decoding process.

[0119] Accordingly, the present disclosure provides a method to improve the efficiency of residual coding and entropy coding by defining one or more residual coding structures and / or one or more entropy coding methods to solve such problems, and by selectively applying at least one of these based on specific syntax elements or parameters.

[0120]

[0121] According to one embodiment of the present disclosure, residual coefficients are encoded or decoded using a residual coding structure, wherein a set of syntactic elements coded within the residual coding structure based on specific syntactic elements or a coding method for the syntactic elements may be optionally applied.

[0122] Specifically, in the encoder, when a specific syntax element is determined, a set of syntax elements or a coding method for the syntax elements can be determined based on the value of the syntax element. Then, residual coefficients can be encoded based on the determined set of syntax elements or the coding method for the syntax elements. In the decoder, when a specific syntax element is obtained from the bitstream, a set of syntax elements or a coding method for the syntax elements can be determined based on the value of the syntax element. Then, residual coefficients can be decoded (obtained) based on the determined set of syntax elements or the coding method for the syntax elements.

[0123] Meanwhile, the fact that the set of syntactic elements coded within the residual coding structure changes means that, while maintaining a single residual coding structure, under specific conditions only some of the total syntactic elements included in the structure are coded, while the remaining elements may not be coded or may be omitted.

[0124] Meanwhile, the fact that the coding method for syntactic elements varies within the residual coding structure means that even when the same syntactic element is coded, the coding-related parameters or tools used to encode or decode that element may be selected differently.

[0125] For example, a change in the coding method may include a change in the context model used to encode or decode syntactic elements within the residual coding structure, and accordingly, different statistical models may be applied to the same syntactic elements.

[0126] As another example, changes in the coding method may include changes in the binary conversion method of syntactic elements. That is, for the same syntactic element, a single-bit based binary conversion method may be used, or a binary conversion method with a multi-bit or hierarchical structure may be optionally applied.

[0127] As another example, changes to the coding method may involve changes to the codeword table. That is, different codeword tables may be selected based on the value distribution or statistical characteristics of syntactic elements, or codewords of different lengths may be mapped to the same value.

[0128] Meanwhile, the aforementioned specific syntax element may be a syntax element defined for changing or selecting a residual coding method. That is, the specific syntax element may be a flag or index defined for the purpose of explicitly indicating a residual coding method.

[0129] Here, specific syntax elements may be defined at a high or low level to explicitly specify the residual coding method. For example, specific syntax elements may be flags or indices defined in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), or slice header (SH).

[0130] As another example, a specific syntax element may be a flag or index defined in a coding tree unit (CTU), coding unit (CU), prediction unit (PU), transform unit (TU), or virtual pipeline data unit (VPDU).

[0131] Meanwhile, the aforementioned specific syntax element may be a syntax element that is not originally defined for the purpose of indicating a residual coding method. That is, the specific syntax element may be a flag or index defined at a high level such as VPS, SPS, PPS, PH, SH, etc., for other purposes.

[0132] For example, a specific syntactic element may be a syntactic element indicating a slice type defined in SH. In this case, the slice type is a syntactic element intended to indicate the encoding and / or decoding characteristics of the slice, but the coding method applied within the residual coding structure may be indirectly determined based on the value of the slice type.

[0133] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to the prediction of the current block or unit. Since the prediction accuracy as well as the statistical distribution and characteristics of the residual signal generated after prediction may vary depending on the prediction method or prediction mode, a set of syntactic elements coded within the residual coding structure or a coding method for the syntactic elements may be selectively applied based on the syntactic element related to the prediction.

[0134] For example, a specific syntactic element may be a syntactic element that indicates whether the current block or unit is predicted by intra-prediction or inter-prediction.

[0135] As another example, a specific syntactic element may be a syntactic element that indicates the prediction mode applied to the current block or unit. In recent video encoding / decoding technologies, various intra-frame prediction modes, inter-frame prediction modes, or prediction modes combined with these are being considered, and the prediction accuracy and residual signal distribution characteristics may differ depending on the prediction mode.

[0136] Table 1 shows examples of various prediction modes considered in recent video encoding / decoding technologies.

[0137]

[0138]

[0139] Depending on the diversity of these prediction modes, a set of syntactic elements or a coding method for syntactic elements coded within the residual coding structure may be selectively applied based on the prediction mode applied to the current block or unit. Meanwhile, Table 1 is merely an example, and the types / names of prediction modes may vary depending on the standard / implementation.

[0140] As another example, a specific syntax element may be a syntax element that indicates whether the prediction mode applied to the current block or unit belongs to one or more predefined prediction mode groups. In this case, each prediction mode group may be configured to include one or more prediction modes. Meanwhile, the prediction mode group may be determined by the encoder and transmitted to the decoder, or determined by an agreement between the encoder and the decoder.

[0141] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to the transform applied to the current block or unit. Since the correlation and distribution characteristics between coefficients of the residual signal may vary depending on the transform method, transform structure, or whether the transform is performed, a set of syntactic elements coded within the residual coding structure or a coding method for the syntactic elements may be selectively applied based on the syntactic element related to the transform.

[0142] For example, a specific syntax element may be a syntax element that indicates whether only a primary transform has been performed on the current block or unit, or whether a secondary transform has been additionally performed after the primary transform.

[0143] As another example, a specific syntax element may be a syntax element that indicates whether to apply a transform skip, where no transformation is performed on the current block or unit.

[0144] As another example, a specific syntactic element may be a syntactic element that indicates whether the transformation applied to the current block or unit is a separable transform or a non-separable transform.

[0145] As another example, a specific syntax element may be a syntax element that indicates the type of transformation or transform kernel applied to the current block or unit.

[0146] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to quantization applied to the current block or unit. Since the correlation and distribution characteristics between coefficients of the residual signal may appear differently depending on the quantization method or quantization parameters, a set of syntactic elements coded within the residual coding structure or a coding method for the syntactic elements may be selectively applied based on the syntactic element related to quantization.

[0147] For example, a specific syntactic element may be a syntactic element that indicates a quantization parameter (QP) applied to the current block or unit.

[0148] Meanwhile, the aforementioned specific syntactic element may include one or more syntactic elements, and if multiple syntactic elements are included, they may be used independently or in combination to determine the set of syntactic elements coded within the residual coding structure and / or the coding method for the syntactic elements.

[0149] Meanwhile, the aforementioned embodiments may be performed independently for each color space component. For example, if YCbCr components exist, a set of syntactic elements or a coding method for syntactic elements coded within a single residual coding structure may be determined for the luminance component Y as described in the aforementioned embodiments, and a set of syntactic elements or a coding method for syntactic elements coded within a single residual coding structure may be determined individually for the chrominance component CbCr as described in the aforementioned embodiments. Alternatively, the methods described in the aforementioned embodiments may be applied individually to the chrominance components Cb and Cr, respectively. Furthermore, the residual coefficients of each component may be coded by applying individual methods to each component.

[0150] Meanwhile, the aforementioned embodiments may be performed independently for each channel. For example, if a YUV channel and an additional channel exist, a set of syntactic elements coded within a single residual coding structure or a coding method for the syntactic elements may be determined for the YUV channel as described in the aforementioned embodiments, and a method as described in the aforementioned embodiments may be individually applied to the additional channel. Alternatively, a set of syntactic elements coded within a single residual coding structure or a coding method for the syntactic elements may be determined for the Y, U, and V channels, respectively, as described in the aforementioned embodiments.

[0151] Meanwhile, the aforementioned embodiment describes a case in which a coding method for a set of syntactic elements or syntactic elements coded within a single residual coding structure is selectively applied based on a specific syntactic element. In addition, according to one embodiment of the present disclosure, a coding method for a set of syntactic elements or syntactic elements coded within a single residual coding structure may be selectively applied based on the size of the current block or unit. Alternatively, a coding method for a set of syntactic elements or syntactic elements coded within a single residual coding structure may be selectively applied based on a syntactic element indicating the size of the current block or unit.

[0152] Here, the size of the current block or unit may refer to any one of the width and height, the product of the width and height, at least one of the width or height, or the sum of the width and height. Generally, regions within an image divided into large blocks or units are homogeneous regions and are likely to have small residuals, while regions divided into small blocks or units are complex regions and are likely to have large residuals. Therefore, depending on the size of the current block or unit, only some syntactic elements may be coded within the residual coding structure in the manner described above, or different context models, binarization, or codeword tables may be selectively applied.

[0153]

[0154] According to one embodiment of the present disclosure, a plurality of residual coding structures are defined, and one of the plurality of residual coding structures may be selectively used based on specific syntactic elements.

[0155] Specifically, in the encoder, when a specific syntax element is determined, one of a plurality of residual coding structures can be determined based on the value of the syntax element. Then, residual coefficients can be encoded based on the determined residual coding syntax structure. In the decoder, when a specific syntax element is obtained from the bitstream, one of a plurality of residual coding structures can be determined based on the value of the syntax element. Then, residual coefficients can be decoded (obtained) based on the residual coding structure.

[0156] Meanwhile, the definition of multiple residual coding structures may mean that multiple residual coding structures can be predefined, wherein at least one of the set of syntactic elements encoded / decoded in relation to the residual, whether the syntactic elements are conditionally included, the signaling order / hierarchical arrangement, and the derivation rules between the syntactic elements differs from one another.

[0157] Meanwhile, the aforementioned specific syntactic element may be a syntactic element defined for the selection of a residual coding structure. That is, the specific syntactic element may be a flag or index defined for the purpose of explicitly indicating which of multiple residual coding structures to use.

[0158] Here, specific syntax elements may be defined at a high level or a low level to explicitly indicate the residual coding structure. For example, specific syntax elements may be flags or indices defined in VPS, SPS, PPS, PH, or SH. As another example, specific syntax elements may be flags or indices defined in CTU, CU, PU, ​​TU, or VPDU.

[0159] Here, in order to construct an arbitrary profile, the value of a specific syntax element may be predefined or limited. For example, when following a specific profile, the specific syntax element may be fixed to at least one of a plurality of possible values, or limited so that only some values ​​are allowed.

[0160] Meanwhile, the aforementioned specific syntax element may be a syntax element that is not originally defined for the purpose of indicating a residual coding structure. That is, the specific syntax element may be a flag or index defined at a high level such as VPS, SPS, PPS, PH, SH, etc., for other purposes.

[0161] For example, a specific syntactic element may be a syntactic element indicating a slice type defined in SH. In this case, the slice type is a syntactic element intended to indicate the encoding and / or decoding characteristics of the slice, but one of a plurality of residual coding structures may be selectively used based on the value of the slice type.

[0162] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to the prediction of the current block or unit. Since the statistical distribution and characteristics of the residual signal generated after prediction may differ depending on the prediction method or prediction mode, one of a plurality of residual coding structures may be selectively used based on the syntactic element related to the prediction.

[0163] For example, a specific syntactic element may be a syntactic element that indicates whether the current block or unit is predicted by intra-prediction or inter-prediction.

[0164] As another example, a specific syntax element may be a syntax element that indicates the prediction mode applied to the current block or unit. As described above in Table 1, depending on the variety of prediction modes, one of a number of residual coding structures may be selectively used based on the prediction mode applied to the current block or unit. Referring to [Table 1], one of a number of residual coding structures may be selectively used based on a syntax element that indicates whether the current block or unit was predicted based on neural network-based intra-prediction (NNIP).

[0165] Additionally, if the prediction mode is predetermined or fixed at the frame or slice level, the same prediction mode may be applied to multiple blocks or units included in the corresponding frame or slice, and accordingly, common residual statistical characteristics may be formed for the entire corresponding frame or slice. In such cases, one of multiple residual coding structures is selected at the frame or slice level based on the prediction mode, and the selected residual coding structure may be consistently applied to the blocks or units included in the corresponding frame or slice.

[0166] As another example, a specific syntax element may be a syntax element that indicates whether the prediction mode applied to the current block or unit belongs to one or more predefined prediction mode groups. In this case, each prediction mode group may be configured to include one or more prediction modes. Meanwhile, the prediction mode group may be determined by the encoder and transmitted to the decoder, or determined by an agreement between the encoder and the decoder.

[0167] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to the transformation applied to the current block or unit. Since the correlation and distribution characteristics between coefficients of the residual signal may vary depending on the transformation method, transformation structure, or whether the transformation is performed, one of a plurality of residual coding structures may be selectively used based on the syntactic element related to the transformation.

[0168] For example, a specific syntax element may be a syntax element that indicates whether only a primary transform has been performed on the current block or unit, or whether a secondary transform has been additionally performed after the primary transform.

[0169] As another example, a specific syntax element may be a syntax element that indicates whether to apply a transform skip, where no transformation is performed on the current block or unit.

[0170] As another example, a specific syntactic element may be a syntactic element that indicates whether the transformation applied to the current block or unit is a separable transform or a non-separable transform.

[0171] As another example, a specific syntax element may be a syntax element that indicates the type of transformation or transform kernel applied to the current block or unit.

[0172] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to quantization applied to the current block or unit. Since the correlation and distribution characteristics between coefficients of the residual signal may appear differently depending on the quantization method or quantization parameters, one of a plurality of residual coding structures may be selectively used based on the syntactic element related to quantization.

[0173] For example, a specific syntactic element may be a syntactic element that indicates a quantization parameter (QP) applied to the current block or unit.

[0174] Meanwhile, the aforementioned specific syntactic element may include one or more syntactic elements, and if multiple syntactic elements are included, they may be used independently or in combination to determine the residual coding structure.

[0175] Meanwhile, the aforementioned embodiment may be performed independently for each color space component. For example, if YCbCr components are present, a residual coding structure may be selected for the luminance component Y in the manner described in the aforementioned embodiment, and a residual coding structure may also be selected individually for the chrominance component CbCr. Alternatively, a residual coding structure may be selected independently for the chrominance components Cb and Cr, respectively. Furthermore, the residual coefficients of each component may be coded based on the individual residual coding structures for each component.

[0176] Meanwhile, the aforementioned embodiment may be performed independently for each channel. For example, if a YUV channel and an additional channel exist, a residual coding structure may be selected for the YUV channel in the manner described in the aforementioned embodiment, and a residual coding structure may also be selected individually for the additional channel. Alternatively, a residual coding structure may be selected independently for each of the Y, U, and V channels.

[0177] Meanwhile, the above-described embodiment explains a case in which one of a plurality of residual coding structures is selectively used based on a specific syntactic element. In addition, according to one embodiment of the present disclosure, one of a plurality of residual coding structures may be selectively used based on the size of the current block or unit. Alternatively, one of a plurality of residual coding structures may be selectively used based on a syntactic element indicating the size of the current block or unit.

[0178] Here, the size of the current block or unit may refer to any one of the width and height, the product of the width and height, at least one of the width or height, or the sum of the width and height. Generally, regions divided into large blocks or units within an image are homogeneous regions and are likely to have small residuals, while regions divided into small blocks or units are complex regions and are likely to have large residuals. Therefore, depending on the size of the current block or unit, one of the multiple residual coding structures may be selectively used in the manner described above.

[0179]

[0180] According to one embodiment of the present disclosure, one or more entropy coding methods may be defined, and one of the one or more entropy coding methods may be selectively applied based on specific syntax elements.

[0181] Specifically, in the encoder, when a specific syntax element is determined, one of a plurality of entropy coding methods may be determined based on the value of the syntax element. Then, entropy encoding may be performed based on the determined entropy coding method. In the decoder, when a specific syntax element is obtained from the bitstream, one of a plurality of entropy coding methods may be determined based on the value of the syntax element. Then, entropy decoding may be performed based on the determined entropy coding method.

[0182] Meanwhile, the fact that entropy coding methods can be applied selectively may mean that the entropy encoding / decoding algorithms themselves applied when including residuals or syntax elements related to residuals in the bitstream or recovering them from the bitstream can be selected differently. For example, one or more entropy coding methods may include at least one of CAVLC (context-based adaptive variable length coding) based on variable length code, CABAC (context-based adaptive binary arithmetic coding) based on arithmetic coding, or PIPE (probability interval partitioning entropy, PIPE), but are not limited thereto.

[0183] Meanwhile, the aforementioned specific syntax element may be a syntax element defined for changing or selecting an entropy coding method. That is, the specific syntax element may be a flag or index defined for the purpose of explicitly indicating which of the multiple entropy coding methods to use.

[0184] Here, specific syntax elements may be defined at a high level or a low level to explicitly indicate the entropy coding method. For example, specific syntax elements may be flags or indices defined in VPS, SPS, PPS, PH, or SH. As another example, specific syntax elements may be flags or indices defined in CTU, CU, PU, ​​TU, or VPDU.

[0185] Here, in order to construct an arbitrary profile, the value of a specific syntax element may be predefined or limited. For example, when following a specific profile, the specific syntax element may be fixed to at least one of a plurality of possible values, or limited so that only some values ​​are allowed.

[0186] Meanwhile, the aforementioned specific syntax element may be a syntax element that is not originally defined for the purpose of indicating the selection of an entropy coding method. That is, the specific syntax element may be a syntax element defined for other purposes, and the entropy coding method applied may be indirectly determined based on the value of said syntax element.

[0187] For example, a specific syntax element may be a syntax element indicating a slice type defined in SH. In this case, the slice type is a syntax element intended to indicate the encoding and / or decoding characteristics of the slice, but one of a plurality of entropy coding methods may be optionally used based on the value of the slice type.

[0188] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to the prediction of the current block or unit. Since the statistical distribution and characteristics of the residual signal generated after prediction may vary depending on the prediction method or prediction mode, one of a plurality of entropy coding methods may be selectively used based on the syntactic element related to the prediction.

[0189] For example, a specific syntactic element may be a syntactic element that indicates whether the current block or unit is predicted by intra-prediction or inter-prediction.

[0190]

[0191] As another example, a specific syntactic element may be a syntactic element that indicates the prediction mode applied to the current block or unit. As described above in Table 1, depending on the variety of prediction modes, one of a plurality of entropy coding methods may be selectively used based on the prediction mode applied to the current block or unit.

[0192] As another example, a specific syntax element may be a syntax element that indicates whether the prediction mode applied to the current block or unit belongs to one or more predefined prediction mode groups. In this case, each prediction mode group may be configured to include one or more prediction modes. Meanwhile, the prediction mode group may be determined by the encoder and transmitted to the decoder, or determined by an agreement between the encoder and the decoder.

[0193] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to the transformation applied to the current block or unit. Since the correlation and distribution characteristics between coefficients of the residual signal may vary depending on the transformation method, transformation structure, or whether the transformation is performed, one of a plurality of entropy coding methods may be selectively used based on the syntactic element related to the transformation.

[0194] For example, a specific syntax element may be a syntax element that indicates whether only a primary transform has been performed on the current block or unit, or whether a secondary transform has been additionally performed after the primary transform.

[0195] As another example, a specific syntax element may be a syntax element that indicates whether to apply a transform skip, where no transformation is performed on the current block or unit.

[0196] As another example, a specific syntactic element may be a syntactic element that indicates whether the transformation applied to the current block or unit is a separable transform or a non-separable transform.

[0197] As another example, a specific syntax element may be a syntax element that indicates the type of transformation or transform kernel applied to the current block or unit.

[0198] Meanwhile, the aforementioned specific syntactic element may be a syntactic element related to quantization applied to the current block or unit. Since the correlation and distribution characteristics between coefficients of the residual signal may vary depending on the quantization method or quantization parameters, one of a plurality of entropy coding methods may be selectively used based on the syntactic element related to quantization.

[0199] For example, a specific syntactic element may be a syntactic element that indicates a quantization parameter (QP) applied to the current block or unit.

[0200] Meanwhile, the aforementioned specific syntactic element may include one or more syntactic elements, and if multiple syntactic elements are included, they may be used independently or in combination to determine the residual coding structure.

[0201] Meanwhile, the aforementioned embodiment describes a case where one of a plurality of entropy coding methods is selectively used based on the value of a specific syntactic element. In addition, according to one embodiment of the present disclosure, one of a plurality of entropy coding methods may be selectively used based on the syntax type to which the syntactic element belongs.

[0202] Here, a syntax type may refer to a higher-level syntax structure containing syntax elements. For example, a video parameter set (VPS) RBSP (raw byte sequence payload) syntax, a sequence parameter set (SPS) RBSP syntax, a picture parameter set (PPS) RBSP syntax, a picture header (PH) syntax, a slice header (SH) syntax, or a slice data (SD) syntax may be defined as different syntax types.

[0203] For example, among the syntax types, an entropy coding method based on arithmetic coding is applied to syntax elements included in slice data (SD) syntax, and an entropy coding method based on variable length code may be applied to syntax elements included in VPS RBSP, SPS RBSP, PPS RBSP, PH, or SH syntax. However, this is not limited thereto, and various combinations of entropy coding methods applied for each syntax type can be configured.

[0204] Meanwhile, the above-described embodiment explains a case where one of a plurality of entropy coding methods is selectively used based on a specific syntactic element. In addition, according to one embodiment of the present disclosure, one of a plurality of entropy coding methods may be selectively used based on the size of the current block or unit. Alternatively, one of a plurality of entropy coding methods may be selectively used based on a syntactic element indicating the size of the current block or unit.

[0205] Here, the size of the current block or unit may refer to any one of the width and height, the product of the width and height, at least one of the width or height, or the sum of the width and height. Generally, regions divided into large blocks or units within an image are homogeneous regions and are likely to have small residuals, while regions divided into small blocks or units are complex regions and are likely to have large residuals. Therefore, depending on the size of the current block or unit, one of a plurality of entropy coding methods may be selectively used in the manner described above.

[0206] Meanwhile, the aforementioned embodiments may be performed independently for each color space component. For example, if YCbCr components are present, an entropy coding method may be selected for the luminance component Y in the manner described in the aforementioned embodiments, and an entropy coding method may also be selected individually for the color difference component CbCr. Alternatively, the method described in the aforementioned embodiments may be applied independently to the color difference components Cb and Cr, respectively.

[0207] Meanwhile, the aforementioned embodiments may be performed independently for each channel. For example, if a YUV channel and an additional channel exist, an entropy coding method may be selected for the YUV channel in the manner described in the aforementioned embodiments, and an entropy coding method may also be selected individually for the additional channels. Alternatively, the methods described in the aforementioned embodiments may be applied independently to the Y, U, and V channels, respectively.

[0208]

[0209] Hereinafter, a parallel processing method based on a plurality of entropy coders according to one embodiment of the present disclosure will be described. Through this method, the speed of entropy encoding and / or entropy decoding can be improved, and a low latency use case of a encoder / decoder including a plurality of entropy coders can be satisfied.

[0210] Here, an entropy coder may refer to an encoding / decoding module that entropy-encodes symbols related to syntax elements or residuals based on the probability of occurrence of the symbols and includes them in a bitstream, or entropy-decodes them from a bitstream and restores them. For example, an entropy coder may be configured to operate based on entropy coding methods such as CABAC (context-based adaptive binary arithmetic coding), CAVLC (context-based adaptive variable length coding), and PIPE (probability interval partitioning entropy).

[0211] For example, in the encoder / decoder, entropy coder A(Entropy coder A, E A ) and entropy coder B(Entropy coder B, E B ) is included, and E A and EB The time required to process one symbol is T respectively. A and T B It can be assumed that.

[0212] In addition, since the residual coding structure, the coding method of residual-related syntactic elements, and / or the entropy coding method may differ according to the embodiments described above, residual-related symbols or syntactic elements to which the same coding structure, coding method, or the same entropy coding method is applied may be classified into a single residual signal group. Accordingly, it can be assumed that residual signal groups RG1, RG2, RG3, RG4, and RG5 exist.

[0213] For example, E A RG2, RG3, and RG5 are assigned to, and E B When RG1 and RG4 are assigned, the total time T required for encoding or decoding the bitstream can be expressed as in Equation 1.

[0214]

[0215]

[0216] In mathematical formula 1, n(RG i ) is the corresponding residual signal group RG i It can mean the number of symbols included in (i=1,2…,5). Therefore, the total time T required for encoding or decoding the bitstream is E A The time and E that process B It can be defined as the sum of the processing times.

[0217] According to one embodiment of the present disclosure, residual signal groups (RG1~RG5) are entropy coders (E) such that the difference in the encoding / decoding time required for each entropy coder is minimized. A , E BIt can be redistributed to ). This allows the decoding and encoding time of the entire bitstream to be significantly reduced.

[0218] Specifically, residual signal groups can be redistributed based on the encoding / decoding time of a unit symbol. Through this, the difference in the time required for each entropy encoder / decoder is minimized, and the entropy encoding / decoding time can be minimized by processing the encoding / decoding of the entire bitstream in parallel.

[0219] Meanwhile, information on which residual signal group to redistribute to multiple entropy coders can be defined at least one of high levels such as VPS, SPS, PPS, PH, SH, or low levels such as CTU, CU, PU, ​​TU.

[0220]

[0221] FIG. 8 is a flowchart illustrating a decoding method according to one embodiment of the present disclosure. The decoding method of FIG. 8 can be performed by an image decoding device.

[0222] The video decoder can obtain one or more syntax elements to determine a residual coding syntax structure from a bitstream (S800).

[0223] Meanwhile, one or more of the above-mentioned syntax elements may include a syntax element indicating whether the current block was predicted based on neural network-based intra prediction (NNIP).

[0224] Meanwhile, the above one or more syntax elements can be obtained from at least one of the picture header (PH) and the slice header (SH).

[0225] Meanwhile, the above one or more syntax elements may include syntax elements indicating whether the current block was predicted based on in-screen prediction and whether it was predicted based on cross-screen prediction.

[0226] Meanwhile, the above one or more syntax elements may include a syntax element indicating at least one of the width and height of the current block.

[0227] Meanwhile, the above one or more syntax elements include a syntax element indicating a transformation method for the current block, and the syntax element indicating the transformation method may include a first transformation syntax element indicating whether to perform a secondary transform and a second transformation syntax element indicating whether the primary transform is a separable transform or a non-separable transform.

[0228] One or more of the above syntax elements can be obtained at a level corresponding to at least one of a Coding tree unit (CTU), Coding unit (CU), Prediction unit (PU), or Transform unit (TU).

[0229] And, the video decoding device can determine one of a plurality of residual coding syntax structures as the residual coding syntax structure for the current block based on the one or more syntax elements (S810).

[0230] Meanwhile, among the plurality of residual coding syntax structures, at least one of the set of syntax elements used to obtain the residual coefficient of the current block, whether the syntax elements are conditionally included, and the signaling order of the syntax elements may differ from one another.

[0231] Meanwhile, the determined residual coding syntax structure above can be used when obtaining residual coefficients for multiple blocks included in the same slice or picture as the current block.

[0232] Meanwhile, the above-mentioned determining step may include the step of determining one of the plurality of residual coding syntax structures as a first residual coding syntax structure for the luminance (luma) component of the current block and the step of determining one of the plurality of residual coding syntax structures as a second residual coding syntax structure for the chroma component of the current block.

[0233] And, the image decoding device can obtain the residual coefficient of the current block based on the determined residual coding syntax structure (S820).

[0234]

[0235] FIG. 9 is a flowchart illustrating an encoding method according to one embodiment of the present disclosure. The encoding method of FIG. 9 can be performed by an image encoding device.

[0236] The video encoding device can determine one or more syntax elements for determining a residual coding syntax structure (S900).

[0237] Meanwhile, the above one or more syntax elements may include a syntax element indicating whether the current block was predicted based on neural network-based intra prediction (NNIP).

[0238] Meanwhile, the above one or more syntax elements may be included in at least one of the picture header (PH) and slice header (SH).

[0239] Meanwhile, the above one or more syntax elements may include syntax elements indicating whether the current block was predicted based on in-screen prediction and whether it was predicted based on cross-screen prediction.

[0240] Meanwhile, the above one or more syntax elements may include a syntax element indicating at least one of the width and height of the current block.

[0241] Meanwhile, the above one or more syntax elements include a syntax element indicating a transformation method for the current block, and the syntax element indicating the transformation method may include a first transformation syntax element indicating whether to perform a secondary transform and a second transformation syntax element indicating whether the primary transform is a separable transform or a non-separable transform.

[0242] Meanwhile, the above one or more syntactic elements may be included in a level corresponding to at least one of a Coding tree unit (CTU), Coding unit (CU), Prediction unit (PU), or Transform unit (TU).

[0243] And, the video encoding device can determine one of a plurality of residual coding syntax structures as the residual coding syntax structure for the current block based on the one or more syntax elements (S910).

[0244] Meanwhile, among the plurality of residual coding syntax structures, at least one of the set of syntax elements used to encode the residual coefficient of the current block, whether the syntax elements are conditionally included, and the signaling order of the syntax elements may differ from one another.

[0245] Meanwhile, the determined residual coding syntax structure above can be used when encoding residual coefficients for multiple blocks included in the same slice or picture as the current block.

[0246] Meanwhile, the above-mentioned determining step may include the step of determining one of the plurality of residual coding syntax structures as a first residual coding syntax structure for the luminance (luma) component of the current block and the step of determining one of the plurality of residual coding syntax structures as a second residual coding syntax structure for the chroma component of the current block.

[0247] Meanwhile, the step of encoding the residual coefficients may include the step of encoding the residual coefficients of the luminance component based on the first residual coding syntax structure and the step of encoding the residual coefficients of the chrominance component based on the second residual coding syntax structure.

[0248] And, the video encoding device can encode the residual coefficient of the current block based on the determined residual coding syntax structure (S920).

[0249] Additionally, a bitstream may be generated by a video encoding method including the steps described in FIG. 9. The bitstream may be stored on a non-transient computer-readable recording medium and may also be transmitted (or streamed).

[0250]

[0251] FIG. 10 is a drawing illustrating an exemplary content streaming system to which an embodiment according to the present disclosure can be applied.

[0252] As illustrated in FIG. 10, a content streaming system to which an embodiment of the present disclosure is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0253] The encoding server described above compresses content input from multimedia input devices, such as smartphones, cameras, and CCTVs, into digital data to generate a bitstream and transmits it to the streaming server. As another example, if multimedia input devices, such as smartphones, cameras, and CCTVs, generate the bitstream directly, the encoding server may be omitted.

[0254] The bitstream may be generated by a video encoding method and / or video encoding device to which an embodiment of the present disclosure is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0255] The streaming server transmits multimedia data to a user device based on a user request through a web server, and the web server can act as a medium to inform the user of available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server can perform the role of controlling commands and responses between each device within the content streaming system.

[0256] The streaming server can receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, to provide a seamless streaming service, the streaming server can store the bitstream for a certain period of time.

[0257] Examples of the above user devices may include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, HMDs (head-mounted displays)), digital TVs, desktop computers, digital signage, etc.

[0258] Each server within the above-mentioned content streaming system can be operated as a distributed server, and in this case, data received from each server can be processed in a distributed manner.

[0259]

[0260] The above embodiments may be performed in the same or a corresponding way in the encoding device and the decoding device. Additionally, an image may be encoded / decoded using at least one of the above embodiments or a combination of at least one.

[0261] The order in which the above embodiments are applied may differ between the encoding device and the decoder. Alternatively, the order in which the above embodiments are applied may be the same between the encoding device and the decoder.

[0262] The above embodiments may be performed for each of the luminance and chrominance signals. Alternatively, the above embodiments for the luminance and chrominance signals may be performed in the same way.

[0263] In the above embodiments, methods are described based on flowcharts as a series of steps or units; however, the present disclosure is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive, other steps may be included, or one or more steps of the flowcharts may be omitted without affecting the scope of the present disclosure.

[0264] The above embodiments may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present disclosure, or they may be those known and available to those skilled in the art of computer software.

[0265] The bitstream generated by the encoding method according to the above embodiment may be stored in a non-transient computer-readable recording medium. Additionally, the bitstream stored in the non-transient computer-readable recording medium may be decoded by the decoding method according to the above embodiment.

[0266] Herein, examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa.

[0267] Although the present disclosure has been described above with specific details such as specific components, limited embodiments, and drawings, this is provided only to aid in a more comprehensive understanding of the present disclosure and is not limited to the above embodiments, and a person skilled in the art to which the present disclosure belongs can make various modifications and variations from this description.

[0268] Accordingly, the scope of the present disclosure is not limited to the embodiments described above, and all things equivalent or equivalently modified to the claims set forth below, as well as the claims set forth below, shall be considered to be within the scope of the scope of the present disclosure.

[0269] The present disclosure may be used in an apparatus for encoding / decoding images and a method for transmitting a bitstream.

Claims

1. A step of obtaining one or more syntax elements for determining a residual coding syntax structure from a bitstream; A step of determining one of a plurality of residual coding syntax structures as the residual coding syntax structure for the current block based on one or more of the above-mentioned syntax elements; and It includes the step of obtaining the residual coefficient of the current block based on the determined residual coding syntax structure, and An image decoding method characterized in that one or more of the above-mentioned syntax elements include a syntax element indicating whether the current block was predicted based on neural network-based intra prediction (NNIP).

2. In Paragraph 1, A video decoding method characterized in that one or more of the above syntax elements are obtained from at least one of a picture header (PH) and a slice header (SH).

3. In Paragraph 1, A video decoding method characterized in that the above plurality of residual coding syntax structures are different from each other in that at least one of the set of syntax elements used to obtain the residual coefficient of the current block, whether the syntax elements are conditionally included, and the signaling order of the syntax elements.

4. In Paragraph 1, A video decoding method characterized in that the above one or more syntactic elements include a syntactic element indicating whether the current block was predicted based on intra-frame prediction and whether it was predicted based on inter-frame prediction.

5. In Paragraph 1, An image decoding method characterized in that the above one or more syntax elements include a syntax element indicating at least one of the width and height of the current block.

6. In Paragraph 1, The above one or more syntax elements include syntax elements that indicate a conversion method for the current block, and An image decoding method characterized in that the syntax element indicating the above transformation method includes a first transformation syntax element indicating whether to perform a secondary transform and a second transformation syntax element indicating whether the primary transform is either a separable transform or a non-separable transform.

7. In Paragraph 1, An image decoding method characterized in that the above-determined residual coding syntax structure is used when obtaining residual coefficients for a plurality of blocks included in the same slice or the same picture as the current block.

8. In Paragraph 1, The above-mentioned determining step is, A step of determining one of the plurality of residual coding syntax structures as a first residual coding syntax structure for the luminance (luma) component of the current block; and A video decoding method comprising the step of determining one of the plurality of residual coding syntax structures as a second residual coding syntax structure for the chroma component of the current block.

9. In Paragraph 8, The step of obtaining the above residual coefficient is, A step of obtaining a residual coefficient of the luminance component based on the first residual coding syntax structure; and An image decoding method comprising the step of obtaining residual coefficients of the chrominance component based on the above second residual coding syntax structure.

10. In Paragraph 1, An image decoding method characterized in that one or more of the above-mentioned syntax elements are obtained at a level corresponding to at least one of a Coding tree unit (CTU), a Coding unit (CU), a Prediction unit (PU), or a Transform unit (TU).

11. A step of determining one or more syntactic elements to determine a residual coding syntax structure; A step of determining one of a plurality of residual coding syntax structures as the residual coding syntax structure for the current block based on one or more of the above-mentioned syntax elements; and It includes the step of encoding the residual coefficients of the current block based on the determined residual coding syntax structure, and An image encoding method characterized in that one or more of the above-mentioned syntax elements include a syntax element indicating whether the current block was predicted based on neural network-based intra prediction (NNIP).

12. In a bitstream transmission method, A step of generating the bitstream based on a video encoding method; and It includes the step of transmitting the above bitstream, The above image encoding method is, A step of determining one or more syntactic elements to determine a residual coding syntax structure; A step of determining one of a plurality of residual coding syntax structures as the residual coding syntax structure for the current block based on one or more of the above-mentioned syntax elements; and It includes the step of encoding the residual coefficients of the current block based on the determined residual coding syntax structure, and A transmission method characterized in that one or more of the above-mentioned syntax elements include a syntax element indicating whether the current block is predicted based on neural network-based intra prediction (NNIP).

Images