Adaptive scanning methods and apparatuses for processing image data based on prediction mode

Abstract

The present disclosure provides scanning methods for processing image data and related apparatuses thereof. The method includes: determining a scan mode based on a prediction mode of the image data and M image blocks in the image data, where the scan mode indicates one or more scan patterns for the M image blocks and a scan order among the M image blocks, and M is an integer greater than 0; and obtaining, based on the scan mode, scanned image data of the image data.

Description

ADAPTIVE SCANNING METHODS AND APPARATUSES FOR PROCESSING IMAGE DATA BASED ON PREDICTION MODETECHNICAL FIELD

[0001] This disclosure relates to the field of video coding, and in particular, to scanning methods for processing image data and related apparatuses thereof.BACKGROUND

[0002] Video coding (encoding and decoding) is widely used in digital video applications, for example, broadcast digital television, video transmission over the Internet and / or mobile networks, real-time conversational applications such as video chat and / or video conferencing, high-density digital versatile discs (DVDs) that are also referred to as BLU-RAY discs, video content capturing and editing systems, or security applications of camcorders.

[0003] In video coding, image data may be processed, such as compressed, packed, stored, or displayed. The image data may be associated with an image frame or an image block within the image frame. During the processing of the image data, two-dimensional (2D) image data may be scanned into one-dimensional (ID) data to improve process efficiency. Conversely, ID data may be recovered to 2D image data according to a corresponding scan manner. How to scan image data effectively and rationally in video coding to improve subsequent process is a problem to be solved.SUMMARY

[0004] This present disclosure provides scanning methods for processing image data and related apparatuses thereof, to scan image data effectively and rationally in video coding to improve subsequent process.

[0005] According to a first aspect, a method for processing image data is described. The method may be applied at an encoder side, for example, an encoder, a module in an encoder, or a circuit or chip (for example, a system on chip (SoC) or a system in package) that is responsible for an encoding function in an encoder. For example, the method is applied to an encoder. The method includes: determining a scan mode based on a prediction mode of the image data and M image blocks in the image data, where the scan mode indicates one or more scan patterns for the M image blocks and a scan order among the M image blocks, and AT is an integer greater than 0; and obtaining, based on the scan mode, scanned image data of the image data.

[0006] According to the scanning method for processing image data described in the first aspect, the encoder determines the scan mode based on the prediction mode of the image dataand the A / image blocks in the image data, where the scan mode indicates the one or more scan patterns for the M image blocks and the scan order among the M image blocks, and M is an integer greater than 0; and encoder obtains, based on the scan mode, scanned image data of the image data. That is, the image data may correspond to one or more scan patterns to represent distribution characteristics of this image data. This method may more accurately match the previous processing of the image data, such as prediction, and more effectively improve the subsequent processing of the image data, such as compression or entropy compression.

[0007] In a possible design, the scan mode further indicates a scan pattern in the one or more scan patterns to be used for each image block in the AT image blocks. That is, for each image block in the M image blocks, a corresponding scan pattern may be indicated. Therefore, data in each image block may be scanned in the more suitable scan pattern.

[0008] In a possible design, the prediction mode includes one or more of: a horizontal prediction for intra prediction; a vertical prediction for intra prediction; a DC prediction for intra prediction; a planar prediction for intra prediction; an angular prediction for intra prediction; or an inter prediction.

[0009] In a possible design, the encoder determines the scan order among the M image blocks; and selects the one or more scan patterns from a set of scan patterns according to the prediction mode of the image data.

[0010] In a possible design, the set of scan patterns may include one or more of: an extended horizontal scan pattern; an extended vertical scan pattern; an extended zigzag scan pattern; or an extended diagonal scan pattern.

[0011] In a possible design, the horizontal scan pattern may be extended or enhanced to obtain the extended horizontal scan pattern. The extended horizontal scan pattern is a combination of a horizontal scan pattern and a diagonal scan pattern. For example, each image block in the AT image blocks is a N*N block, N is an integer equal to or greater than 4, and the extended horizontal scan pattern includes: the horizontal scan pattern of scanning part or all elements in N rows in the N*N block; and the diagonal scan pattern of scanning remaining elements in the A'* / block. In an example, N / 4 ov Nil or 27V / 3 rows are scanned in the horizontal direction and the remaining rows are scanned in an interspersed manner using both horizontal and diagonal directions. In another example, the first row or the last row is scanned in the horizontal direction, part elements in the remaining rows are scanned in the horizontal direction, and the remaining elements are scanned in the diagonal direction.

[0012] In a possible design, when N equals to 4, and the extended horizontal scan pattern, includes: scanning starting from an element in a first row and a first column, to an element inthe first row and a second column, to an element in the first row and a third column, to an element in the first row and a fourth column; to an element in a second row and the first column, to an element in the second row and the second column, to an element in the second row and the third column; to an element in a third row and the first column, to an element in the second row and the fourth column; to an element in the third row and the second column, to an element in the third row and the third column; to an element in a fourth row and the first column, to an element in the third row and the fourth column; to an element in the fourth row and the second column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

[0013] In the above design, the 4*4 block may be used as the basic processing unit. For the hardware architecture of the encoder, there is a problem how friendly the encoder’s algorithm is for the hardware architecture of the encoder. For some chips, a lower amount of data processing is preferable. For example, scanning larger blocks of a size 8x8 as a whole may result in a loss of performance compared to blocks of a size 4x4. In this situation, an algorithm oriented toward 4x4 blocks may potentially have an advantage on a specific chip compared to one that needs simultaneous access to data in 8x8 blocks. For example, 4x4 blocks may provide improving of BD-rate on 0.2-0.6% compared to H.264 scanning schemes.

[0014] In a possible design, the vertical scan pattern may be extended or enhanced to obtain the extended vertical scan pattern. The extended vertical scan pattern is a combination of a vertical scan pattern and a diagonal scan pattern. For example, each image block in the AT image blocks is a 7V*7Vblock, N is an integer equal to or greater than 4, and the extended vertical scan pattern includes: the vertical scan pattern of scanning part or all elements in N columns in the N*N block; and the diagonal scan pattern of scanning remaining elements in the N*N block. In an example, N / 4 o N / 2 or 27V / 3 columns are scanned in the vertical direction and the remaining columns are scanned in an interspersed manner using both vertical and diagonal directions. In another example, the first column or the last column is scanned in the vertical direction, part elements in the remaining columns are scanned in the vertical direction, and the remaining elements are scanned in the diagonal direction.

[0015] In a possible design, when TV equals to 4, and the extended vertical scan pattern, includes: scanning starting from an element in a first row and a first column, to an element in a second row and the first column, to an element in a third row and the first column, to an element in a fourth row and the first column; to an element in the first row and a second column, to an element in the second row and the second column, to an element in the third row and the second column; to an element in the first row and a third column, to an element in the fourth row andthe second column; to an element in the second row and the third column, to an element in the third row and the third column; to an element in the first row and the fourth column, to an element in the fourth row and the third column; to an element in the second row and the fourth column, to an element in the third row and the fourth column; and to element in the fourth row and the fourth column.

[0016] Similar to the extended horizontal scan pattern, using this extended vertical scan pattern, a trade-off between compression performance and hardware efficiency may also be optimal.

[0017] In a possible design, the zigzag scan pattern may be extended or enhanced to obtain the extended zigzag scan pattern. The extended zigzag scan pattern is a combination of a raster scan pattern and a zigzag scan pattern. For example, each image block in the M image blocks is a N*N block, N is an integer equal to or greater than 4, and the extended zigzag scan pattern includes: the raster scan pattern of scanning starting from an element in a first row and a first column in the N*N block to an element in a second row and a second column in the N*N block; and the zigzag scan pattern of scanning remaining elements in theblock, where the zigzag scan pattern follows a scanning trend of the raster scan pattern. That is, regions in the image data, which are located at the upper left in the image data, may be scanned by the raster scan pattern, and the remaining regions in the image data may be scanned by the zigzag scan pattern. This scan pattern may be more suitable to other predictions in addition to the horizontal prediction for intra prediction and the vertical prediction for intra prediction.

[0018] In a possible design, when TV equals to 4, and the extended zigzag scan pattern, includes: scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in a second row and the first column, to an element in the second row and the second column; to an element in a third row and the first column, to an element in the first row and a third column; to an element in the first row and a fourth column, to an element in the second row and the third column, to an element in the third row and the second column, to an element in a fourth row and the first column, to an element in the fourth row and the second column; to an element in the third row and the third column, to an element in the second row and the fourth column, to an element in the third row and the fourth column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

[0019] Similar to the extended horizontal or vertical scan pattern, using this extended zigzag scan pattern, a trade-off between compression performance and hardware efficiency may also be optimal.

[0020] In a possible design, the diagonal scan pattern may be extended or enhanced to obtain the extended diagonal scan pattern. The extended diagonal scan pattern is a combination of a raster scan pattern and a diagonal scan pattern. For example, each image block in the A / image blocks is a N*N block, N is an integer equal to or greater than 4, and the extended diagonal scan pattern includes: the raster scan pattern of scanning starting from an element in a first row and a first column in the N*N block to an element in a second row and a second column in the N*N block; and the diagonal scan pattern of scanning remaining elements in the N*N block in the diagonal scan pattern, where the diagonal scan pattern follows a scanning trend of the raster scan pattern. That is, regions in the image data, which are located at the upper left or at the lower right in the image data, may be scanned by the raster scan pattern, and the remaining regions in the image data may be scanned by the diagonal scan pattern. This scan pattern may be more suitable to the prediction for intra prediction in addition to the horizontal prediction for intra prediction and the vertical prediction for intra prediction.

[0021] In a possible design, when N equals to 4, and the extended diagonal scan pattern, includes: scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in a second row and the first column, to an element in the second row and the second column; to an element in the first row and a third column, to an element in a third row and the first column; to an element in the first row and a fourth column, to an element in the second row and the third column, to an element in the third row and the second column, to an element in a fourth row and the first column; to an element in the second row and the fourth column, to an element in the third row and the third column, to an element in the fourth row and the second column; and to an element in the third row and the fourth column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

[0022] Similar to the extended horizontal or vertical pattern or zigzag scan pattern, using this extended diagonal scan pattern, a trade-off between compression performance and hardware efficiency may also be optimal.

[0023] In a possible design, there is one 4*4 block, when the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern is used. When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern is used. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended zigzag scan pattern is used, or the extended diagonal scan pattern is used.

[0024] In a possible design, the scan order among the M image blocks is from an upper left image block first and a lower right image block last.

[0025] In a possible design, there are 24*4 blocks.

[0026] In a possible design, the 2 4*4 blocks are arranged horizontally. When the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern is used and the scanning on the 24*4 blocks is horizontal. When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern is used and the scanning on the 2 4*4 blocks is horizontal. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended zigzag scan pattern is used and the scanning on the 2 4*4 blocks is horizontal. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended diagonal scan pattern is used and the scanning on the 24*4 blocks is horizontal.

[0027] In a possible design, the 24*4 blocks are arranged vertically. When the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern is used and the scanning on the 2 4*4 blocks is vertical. When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern is used and the scanning on the 24*4 blocks is vertical. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended zigzag scan pattern is used and the scanning on the 2 4*4 blocks is vertical. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended diagonal scan pattern is used and the scanning on the 24*4 blocks is vertical.

[0028] In a possible design, there are 44*4 blocks.

[0029] In a possible design, the scan mode indicates one extended horizontal scan pattern or one extended vertical horizontal scan pattern, the scan order is from the upper left image block, to the lower left image block, to the upper right image block, and to the lower right image block.

[0030] In a possible design, the scan mode indicates one extended horizontal scan pattern or one extended vertical horizontal scan pattern, the scan order is from an upper left image block, to an upper right image block, to a lower left image block, and to a lower right image block.

[0031] In a possible design, in a case where AT equals to 4 and the scan mode indicates hybrid scan patterns, the scan order is from an upper left image block using an extended zigzag scanpattern, to a lower left image block using an extended horizontal scan pattern, to an upper right image block using an extended vertical scan pattern, to a lower right image block using the extended zigzag scan pattern.

[0032] In a possible design, in a case where AT equals to 4 and the scan mode indicates hybrid scan patterns, the scan order is from an upper left image block using an extended zigzag scan pattern, to an upper right image block using an extended vertical scan pattern, to a lower left image block using an extended horizontal scan pattern, and to a lower right image block using the extended zigzag scan pattern.

[0033] According to a second aspect, a method for processing image data is described. The method may be applied at a decoder side, for example, a decoder, a module in a decoder, or a circuit or chip (for example, a system on chip (SoC) or a system in package) that is responsible for a decoding function in a decoder. For example, the method is applied to a decoder. The method includes: obtaining scanned image data and a scan mode, wherein the scan mode is determined based on a prediction mode of the image data and AT image blocks in the image data, wherein the scan mode indicates one or more scan patterns for the AT image blocks and a scan order among the A / image blocks, andA / is an integer greater than 0; and recovering, based on an inverse of the scan mode, the image data from the scanned image data.

[0034] The same or similar possible designs in the second aspect may refer to the related description in the first aspect.

[0035] According to a third aspect, an apparatus for processing image data is described. The apparatus has a function of implementing the first aspect or any possible design in the first aspect. For example, the apparatus includes a corresponding module, unit, or means for performing operations in the first aspect or any possible design in the first aspect. The module, unit, or means may be specifically implemented by using software, may be implemented by using hardware, or may be implemented by using software in combination with hardware.

[0036] According to a fourth aspect, an apparatus for processing image data is described. The apparatus has a function of implementing the second aspect or any possible design in the second aspect. For example, the apparatus includes a corresponding module, unit, or means for performing operations in the second aspect or any possible design in the second aspect. The module, unit, or means may be specifically implemented by using software, may be implemented by using hardware, or may be implemented by using software in combination with hardware.

[0037] According to a fifth aspect, another apparatus for processing image data is described. The apparatus includes a memory and one or more processors. The memory is configured tostore a part or all of a necessary computer program or instructions for implementing a function in the first aspect or any possible design in the first aspect. The one or more processors may execute the computer program or the instructions, and when the computer program or the instructions is / are executed, the apparatus is enabled to implement the method in the first aspect or any possible design in the first aspect.

[0038] According to a sixth aspect, another apparatus for processing image data is described. The apparatus includes a memory and one or more processors. The memory is configured to store a part or all of a necessary computer program or instructions for implementing a function in the second aspect or any possible design in the second aspect. The one or more processors may execute the computer program or the instructions, and when the computer program or the instructions is / are executed, the apparatus is enabled to implement the method in the second aspect or any possible design in the second aspect.

[0039] According to a seventh aspect, a system is described. The system includes an apparatus for implementing the first aspect or any possible design in the first aspect and an apparatus for implementing the second aspect or any possible design in the second aspect.

[0040] According to an eighth aspect, a computer-readable storage medium is described. The computer-readable storage medium stores computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer is enabled to perform the method in any one of the possible designs of the first aspect to the second aspect.

[0041] According to a ninth aspect, a computer program product is described. When a computer reads and executes the computer program product, the computer is enabled to perform the method in any one of the possible designs of the first aspect to the second aspect.BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To describe technical solutions in some embodiments of the present disclosure more clearly, the following describes accompanying drawings for describing embodiments of the present disclosure.

[0043] FIG. 1 A is a block diagram of an example of a video coding system for implementing one or more embodiments of the present disclosure;

[0044] FIG. IB is a block diagram of an example of another video coding system for implementing one or more embodiments of the present disclosure;

[0045] FIG. 2 is a block diagram of an example structure of an encoder for implementing one or more embodiments of the present disclosure;

[0046] FIG. 3 is a block diagram of an example structure of a decoder for implementing one or more embodiments of the present disclosure;

[0047] FIG. 4 is a block diagram of an example structure of an encoder for implementing one or more embodiments of the present disclosure;

[0048] FIG. 5 is a schematic diagram of an example of subfigure partitioning for implementing one or more embodiments of the present disclosure;

[0049] FIG. 6 is a schematic diagram of an example of a wavelet transform for implementing one or more embodiments of the present disclosure;

[0050] FIG. 7A is a schematic diagram of an example structure of a decoder for implementing one or more embodiments of the present disclosure;

[0051] FIG. 7B is a schematic diagram of another example structure of a decoder for implementing one or more embodiments of the present disclosure;

[0052] FIG. 8 is a schematic diagram of a structure of an example of a figure bitstream for implementing one or more embodiments of the present disclosure;

[0053] FIG. 9 is a block diagram of an example of a video coding device for implementing one or more embodiments of the present disclosure;

[0054] FIG. 10 is a block diagram of an example of another video coding device for implementing one or more embodiments of the present disclosure;

[0055] FIG. 11 is a schematic illustration of an example scheme of video codecs according to one or more embodiments of the present disclosure;

[0056] FIG. 12A is a schematic illustration of an example zigzag pattern according to one or more embodiments of the present disclosure;

[0057] FIG. 12B is a schematic illustration of an example field pattern according to one or more embodiments of the present disclosure;

[0058] FIG. 12C is a schematic illustration of an example raster-scan pattern according to one or more embodiments of the present disclosure;

[0059] FIG. 13 A is a schematic illustration of an example vertical scan pattern according to one or more embodiments of the present disclosure;

[0060] FIG. 13B is a schematic illustration of an example horizontal scan pattern according to one or more embodiments of the present disclosure;

[0061] FIG. 13C is a schematic illustration of an example diagonal up-right scan pattern according to one or more embodiments of the present disclosure;

[0062] FIG. 14 is a flowchart of an example scanning method for processing image data according to one or more embodiments of the present disclosure;

[0063] FIG. 15A is a schematic illustration of an example extended horizontal scan pattern according to one or more embodiments of the present disclosure;

[0064] FIG. 15B is a schematic illustration of an example extended vertical scan pattern according to one or more embodiments of the present disclosure;

[0065] FIG. 15C is a schematic illustration of an example extended zigzag scan pattern according to one or more embodiments of the present disclosure;

[0066] FIG. 15D is a schematic illustration of an example extended diagonal scan pattern according to one or more embodiments of the present disclosure;

[0067] FIGS. 16A-16J are schematic illustrations of scan modes for 2 image blocks according to one or more embodiments of the present disclosure;

[0068] FIGS. 17A-17J are other schematic illustrations of scan modes for 2 image blocks according to one or more embodiments of the present disclosure;

[0069] FIGS. 18A-18H are schematic illustrations of scan modes for 4 image blocks according to one or more embodiments of the present disclosure;

[0070] FIG. 19 is a flowchart of another example scanning method for processing image data according to one or more embodiments of the present disclosure; and

[0071] FIG. 20 is a schematic block of an example scanning apparatus for processing image data according to one or more embodiments of the present disclosure.DETAILED DESCRIPTION

[0072] The following describes embodiments of the present disclosure with reference to the accompanying drawings in embodiments of the present disclosure. In the following description, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used are shown by the accompanying drawings.

[0073] In embodiments of the present disclosure, “at least one” means one or more, and “a plurality of’ means two or more. “And / or” describes an association relationship between associated objects, and represents that three relationships may exist. For example, A and / or B may represent the following cases: only A exists, both A and B exist, and only B exists, where A and B may be singular or plural. The character “ / ” generally indicates an “or” relationship between the associated objects. “At least one item (piece) of the following” or a similar expression thereof means any combination of these items, including a singular item (piece) or any combination of plural items (pieces). For example, at least one of a, b, or c may indicate: a; b; c; a and b; a and c; b and c; or a, b, and c, where a, b, and c may be singular or plural.

[0074] Technical solutions described in some embodiments of the present disclosure may be applicable to digital video applications, for example, broadcast digital television, video transmission over the Internet and / or mobile networks, real-time conversational applications such as video chat and / or video conferencing, high-density digital versatile discs (DVDs), also referred to as Blu-ray discs, video content capturing and editing systems, or security applications of camcorders.

[0075] Technical solutions described in some embodiments of the present disclosure may be applicable to video coding standards. The video coding standards may include, but are not limited to moving picture experts group 1 (MPEG-1), MPEG-2, MPEG-4, international telecommunication union telecommunication standardization sector (ITU-T) H.262 / MPEG-2, H.263, H.264 / advanced video coding (AVC), H.265 / high efficiency video coding (HEVC), H.266 / versatile video coding (VVC), video processing 8 (VP8), VP9, audio video coding standard (AVS), AOMedia video 1 (AVI), or extensions of these standards, for example, scalability and / or 3D (three-dimensional) extensions. Technical solutions described in some embodiments of the present disclosure may also be applicable to future or next video coding standards.

[0076] Terms used in embodiments of the present disclosure are merely used to explain specific embodiments of the present disclosure, but are not intended to limit the present disclosure. The following first briefly describes some concepts that may be mentioned in embodiments of the present disclosure. These concepts are introduced solely to explain embodiments of the present disclosure and are not intended to limit the scope of the present disclosure.

[0077] Residual: A difference between a reconstructed value and a predicted value of a sample or data element.

[0078] Residual Block: An M*N matrix (or array) of residuals corresponding to a coding block, where M and N are integers greater than or equal to 1.

[0079] Dequantization: A process of obtaining reconstructed residuals by scaling quantized residuals.

[0080] Partition: A division of a set into subsets, where each element of the set belongs to and is only in one subset.

[0081] Partition Type: An organizational method of subsets obtained through partitioning.

[0082] Decoded Picture: An image reconstructed by a decoder based on a bitstream.

[0083] Prediction: A specific implementation of a prediction process.

[0084] Prediction Process: A process of obtaining predicted values of current samples using previously decoded samples.

[0085] Syntax Element: A result obtained after parsing a data unit in the bitstream.

[0086] Bitstream: A binary data stream formed by all or part of samples of an encoded image.

[0087] Video coding usually refers to processing of a sequence of pictures, where the sequence of pictures forms a video or a video sequence. In the field of video coding, the terms “picture”, “frame”, or “image” may be used as synonyms or interchangeably. Video coding used in this specification indicates video encoding or video decoding. Video encoding is performed on a source side and usually includes processing (for example, compressing) an original video picture to reduce an amount of data for representing the video picture, thereby enabling more efficient storage and / or transmission. Video decoding is performed on a destination side and usually includes inverse processing relative to the source side, to reconstruct the video picture. Video picture “coding” in embodiments is understood as “encoding” or “decoding” of a video sequence. A combination of an encoding part and a decoding part is also referred to as coding (encoding and decoding).

[0088] A video sequence includes a series of pictures, a picture is further partitioned into slices, and a slice is further partitioned into blocks. In video coding, coding processing is performed on a per-block basis. In some video coding standards, a concept “block” is further extended. For example, a macroblock (MB) is introduced in the H.264 standard. The macroblock may be further partitioned into a plurality of prediction blocks for predictive coding. In the HEVC standard, a plurality of block units are functionally classified by using basic concepts such as a coding unit (CU), a prediction unit (PU), and a transform unit (TU). The plurality of block units are described by employing a tree-based structure. For example, a CU may be partitioned into smaller CUs based on a quadtree, and a smaller CU may continue to be partitioned to generate a quadtree structure. The CU may correspond to a coding block and is a basic unit for partitioning and coding a to-be-coded image. A PU and a TU also have a similar tree structure. The PU may correspond to a prediction block and is a basic unit of predictive coding. The PU is further partitioned into a plurality of PUs in a partition mode. The TU may correspond to a transform block and is a basic unit for transforming a prediction residual. However, in essence, all of the CU, the PU, and the TU are conceptually blocks (or image blocks). For a concept of image data in embodiments of this disclosure may include any of blocks described herein and refer to description herein for details.

[0089] In this specification, for ease of description and understanding, a to-be-processed image block in a current coding image may be referred to as a current block, for example, in encoding,the to-be-processed image block refers to a block currently being encoded, and in decoding, the to-be-processed image block refers to a block currently being decoded. A decoded image block that is in a reference image and that is for predicting the current block is referred to as a reference block. In other words, the reference block is a block that provides a reference signal for the current block, where the reference signal indicates a pixel value, a sampling value, or a sampling signal in the image block. A block that provides a prediction signal for the current block in a reference image may be referred to as a prediction block. The prediction signal indicates a pixel value, a sampling value, or a sampling signal in the prediction block. For example, after a plurality of reference blocks are traversed, an optimal reference block is found, and the optimal reference block will provide prediction for the current block, and may be referred to as a prediction block.

[0090] In a case of lossless video coding, original video pictures can be reconstructed. In other words, reconstructed video pictures have same quality as the original video pictures (assuming that no transmission loss or other data loss occurs during storage or transmission). In a case of lossy video coding, further compression is performed through, for example, quantization, to reduce an amount of data for representing video pictures, and the video pictures may not be completely reconstructed on a decoder side. In other words, quality of reconstructed video pictures is lower or poorer than that of the original video pictures.

[0091] The following describes a system architecture to which embodiments of the present disclosure are applicable. Modules / components shown in block diagrams (or schematic diagrams, conceptual diagrams, structural diagrams, or system diagrams) are examples of the present disclosure. The actual block diagrams (or schematic diagrams, conceptual diagrams, structural diagrams, or system diagrams) may include more or fewer modules / components than those shown in the diagrams, or may have different component configurations. Moreover, various components / modules shown in the diagrams may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing units and / or application-specific integrated circuits.

[0092] FIG. 1 A is a schematic block diagram of an example of a video encoding and decoding system 10 to which one or more embodiments of the present disclosure is applicable. As shown in FIG. 1 A, the video encoding and decoding system 10 may include a source device 12 and a destination device 14. The source device 12 generates encoded video data, and therefore the source device 12 may be referred to as a video encoding apparatus. The destination device 14 may decode the encoded video data generated by the source device 12, and therefore the destination device 14 may be referred to as a video decoding apparatus. In variousimplementation solutions, the source device 12, the destination device 14, or both the source device 12 and the destination device 14 may include one or more processors and a memory coupled to the one or more processors. The memory may include but is not limited to a randomaccess memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or any other medium that can be used to store desired program code in a form of instructions or a data structure accessible to a computer, as described in this specification. The source device 12 and the destination device 14 may include various apparatuses, including a desktop computer, a mobile computing apparatus, a notebook (for example, a laptop) computer, a tablet computer, a set-top box, a telephone handset such as a “smart” phone, a television, a camera, a display apparatus, a digital media player, a video game console, a vehicle-mounted computer, a wireless communication device, or the like.

[0093] Although FIG. 1 A depicts the source device 12 and the destination device 14 as separate devices, in one or more device embodiments, both the source device 12 and the destination device 14 or functionalities of both the source device 12 and the destination device 14, that is, the source device 12 or a corresponding functionality and the destination device 14 or a corresponding functionality, may alternatively be included in a single device. In such one or more embodiments, the source device 12 or the corresponding functionality and the destination device 14 or the corresponding functionality may be implemented by using same hardware and / or software or by using separate hardware and / or software or any combination thereof.

[0094] A communication connection between the source device 12 and the destination device 14 may be implemented through a link 13, and the destination device 14 may receive encoded video data from the source device 12 through the link 13. The link 13 may include one or more media or apparatuses capable of moving the encoded video data from the source device 12 to the destination device 14. In an example, the link 13 may include one or more communication media that enable the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. In this example, the source device 12 may modulate the encoded video data according to a communication standard (for example, a wireless communication protocol), and may transmit modulated video data to the destination device 14. The one or more communication media may include a wireless communication medium and / or a wired communication medium, for example, a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form a part of a packet-based network, and the packet-based network is, for example, a local area network, a wide area network, or a global network (for example, the Internet). The one or morecommunication media may include a router, a switch, a base station, or another device that facilitates communication from the source device 12 to the destination device 14.

[0095] The source device 12 includes an encoder 20. For example, the source device 12 may further include a picture source 16, a picture preprocessor 18, and a communication interface 22. In a specific implementation form, the encoder 20, the picture source 16, the picture preprocessor 18, and the communication interface 22 may be hardware components in the source device 12, or may be software programs in the source device 12. The following describes the components separately.

[0096] The picture source 16 may include or be any type of picture capturing device configured to, for example, capture a real-world picture, and / or any type of device for generating a picture or comment (for screen content encoding, some text on a screen is also considered as a part of a to-be-encoded picture or image), for example, a computer graphics processor configured to generate a computer animation picture, and / or any type of device configured to obtain and / or provide a real-world picture or a computer animation picture (for example, screen content or a virtual reality (VR) picture), and / or any combination thereof (for example, an augmented reality (AR) picture). The picture source 16 may be a camera configured to capture a picture or a memory configured to store a picture. The picture source 16 may further include any type of (internal or external) interface through which a previously captured or generated picture is stored and / or a picture is obtained or received. When the picture source 16 is a camera, the picture source 16 may be, for example, a local camera, or an integrated camera integrated into the source device. When the picture source 16 is a memory, the picture source 16 may be a local memory or, for example, an integrated memory integrated into the source device. When the picture source 16 includes an interface, the interface may be, for example, an external interface for receiving a picture from an external video source. The external video source is, for example, an external picture capturing device such as a camera, an external memory, or an external picture generation device. The external picture generation device is, for example, an external computer graphics processor, a computer, or a server. The interface may be any type of interface, for example, a wired or wireless interface or an optical interface, according to any proprietary or standardized interface protocol.

[0097] A picture may be considered as a two-dimensional (2D) array or matrix of picture elements. A pixel in the array may also be referred to as a sample. A quantity of samples in horizontal and vertical directions (or axes) of the array or the picture defines a size and / or resolution of the picture. For representation of a color, three color components are usually employed. To be specific, the picture may be represented as or include three sample arrays. Forexample, in a red, green, and blue (RGB) format or color space, a picture includes a corresponding red, green, and blue sample array. However, in video coding, each sample is usually represented in a luminance / chrominance format or color space. For example, a picture in a YUV format includes a luminance component indicated by Y (or sometimes L) and two chrominance components indicated by U and V. The luminance (luma) component Y represents luminance or gray level intensity (for example, both are the same in a gray-scale picture), and the two chrominance (chroma) components U and V represent chrominance or color information components. Accordingly, the picture in the YUV format includes a luminance sample array of luminance sample values (Y) and two chrominance sample arrays of chrominance values (U and V). A picture in an RGB format may be transformed or converted to a picture in a YUV format and vice versa. This process is also referred to as color conversion or transform. If a picture is monochrome, the picture may include only a luminance sample array. In one or more embodiments of the present disclosure, a picture transmitted by the picture source 16 to the picture processor such as a picture preprocessor 18 may also be referred to as raw picture data 17.

[0098] The picture preprocessor 18 is configured to receive raw picture data 17 and perform preprocessing on the raw picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. For example, the picture preprocessor 18 may perform preprocessing: trimming, color format transformation (for example, from the RGB format to the YUV format), color correction, denoising, or the like.

[0099] The encoder 20 is configured to receive the preprocessed picture data 19, and process the preprocessed picture data 19 in related prediction mode, and / or transform manner, and / or quantization manner, and / or compress manner to provide the encoded picture data 21 (the following further describes structural details of the encoder 20 based on FIG. 2, FIG. 4, or FIG.5). In some embodiments, the encoder 20 may be configured to perform embodiments described below, to implement scanning methods for processing image data in described in the present disclosure.

[0100] The communication interface 22 may be configured to receive the encoded picture data 21 from the encoder directly or indirectly, and transmit the encoded picture data 21 to the destination device 14 or any other device (for example, a memory) through the link 13 for storage or direct reconstruction. The any other device may be any device configured for decoding or storage. The communication interface 22 may be, for example configured to encapsulate the encoded picture data 21 into an appropriate format, for example, a data packet, for transmission through the link 13.

[0101] The destination device 14 includes a decoder 30. For example, the destination device 14 may further include a communication interface 28, a picture post-processor 32, and a display device 34. The following describes the components separately.

[0102] The communication interface 28 may be configured to receive the encoded picture data 21 from the source device 12 or any other source. The any other source is, for example, a storage device. The storage device is, for example, an encoded picture data storage device. The communication interface 28 may be configured to transmit or receive the encoded picture data 21 through the link 13 between the source device 12 and the destination device 14 or through any type of network. The link 13 is, for example, a direct wired or wireless connection, and the any type of network is, for example, a wired or wireless network or any combination thereof, or any type of private or public network, or any combination thereof. The communication interface 28 may be, for example configured to decapsulate the data packet transmitted through the communication interface 22, to obtain the encoded picture data 21.

[0103] Both the communication interface 28 and the communication interface 22 may be configured as unidirectional communication interfaces or bidirectional communication interfaces, and may be configured to, for example, transmit and receive messages to establish a connection, and acknowledge and exchange any other information related to a communication link and / or data transmission such as encoded picture data transmission.

[0104] The decoder 30 is configured to receive the encoded picture data 21 and provide the decoded picture data 31 or the decoded picture 31 (the following further describes structural details of the decoder 30 based on FIG. 3, FIG. 4, or FIG. 5). In some embodiments, the decoder 30 may be configured to perform embodiments described below, to implement scanning methods for processing image data described in the present disclosure.

[0105] The picture post-processor 32 is configured to post-process the decoded picture data 31 (also referred to as reconstructed picture data) to obtain post-processed picture data 33. The picture post-processor 32 may perform post-processing of color format transformation (for example, from a YUV format to an RGB format), color correction, trimming, re-sampling, or any other processing, and may be further configured to transmit the post-processed picture data 33 to the display device 34.

[0106] The display device 34 is configured to receive the post-processed picture data 33 to display a picture, for example, to a user or a viewer. The display device 34 may be or include any type of display configured to present a reconstructed picture, for example, an integrated or external display or monitor. For example, the display may include a liquid-crystal display (LCD), an organic light-emitting diode (LED) (OLED) display, a plasma display, a projector,a micro-LED display, a liquid-crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0107] As will be apparent for a person skilled in the art based on the description, existence and (accurate) division of functionalities of the different units or functionalities of the source device 12 and / or the destination device 14 shown in FIG. 1 A may vary with an actual device and application. The source device 12 and the destination device 14 each may include any one of a wide range of devices, including any type of handheld or stationary device, for example, a notebook or laptop computer, a mobile phone, a smartphone, a pad or a tablet computer, a video camera, a desktop computer, a set-top box, a television, a camera, a vehicle-mounted device, a display device, a digital media player, a video game console, a video streaming transmission device (such as a content service server or a content distribution server), a broadcast receiver device, and a broadcast transmitter device, and may not use or may use any type of operating system.

[0108] The encoder 20 and the decoder 30 each may be implemented as any one of various appropriate circuits, for example, one or more microprocessors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), discrete logic, hardware, or any combinations thereof. If the technologies are implemented partially by using software, a device may store software instructions in an appropriate and non-transitory computer-readable storage medium and may execute instructions by using hardware such as one or more processors, to perform the technologies of this disclosure. Any of the foregoing content (including hardware, software, a combination of hardware and software, and the like) may be considered as one or more processors.

[0109] In some cases, the video encoding and decoding system 10 shown in FIG. 1 A is merely an example, and the technologies of the present disclosure may be applied to a video coding setting (for example, video encoding or video decoding) that does not need to include any data communication between encoding and decoding devices. In another example, data may be retrieved from a local memory, streamed over a network, or the like. A video encoding device may encode data and store data into the memory, and / or a video decoding device may retrieve and decode data from the memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into the memory and / or retrieve data from the memory and decode the data.

[0110] It is understood that the encoder 20 may be deployed on a terminal device or a server on a cloud side, the decoder 30 may be deployed on a terminal device or a server on a cloudside, or the encoder 20 and the decoder 30 may be jointly deployed on a terminal device or a server on a cloud side.

[0111] In a scenario, the encoder 20 and the decoder 30 may be deployed on a terminal device. The encoder 20 may perform coding compression on a video on the terminal device, perform secondary compression (or incremental storage compression) in the video encoding method provided in embodiments of this disclosure, and store compressed data. When the video needs to be played back, the stored compressed data may be decoded.

[0112] In a scenario, the encoder 20 and the decoder 30 may be deployed on a plurality of terminal devices. The encoder 20 may perform coding compression on a video on the terminal device, perform secondary compression (or incremental storage compression) in the video encoding method provided in embodiments of this disclosure, and transmit compressed data to another terminal device. When the other device needs to play back the video, the other device may decode the stored compressed data.

[0113] In a scenario, the encoder 20 and the decoder 30 may be deployed on a terminal device and a server on a cloud side. The encoder 20 may perform coding compression on a video on the terminal device, perform secondary compression (or incremental storage compression) in the video encoding method provided in embodiments of this disclosure, and transmit compressed data to the server.

[0114] FIG. IB is an illustrative diagram of an example of a video coding system 10B including the encoder 20 in FIG. 2 and / or the decoder 30 in FIG. 3 according to one or more example embodiments. The video coding system 10B can implement a combination of various technologies in embodiments of the present disclosure. In the illustrated implementations, the video coding system 10B may include an imaging device 41, the encoder 20, the decoder 30 (and / or a video encoder / decoder implemented by using a logic circuit (not shown) of a processing unit 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0115] As shown in FIG. IB, the imaging device 41, the antenna 42, the processing unit 46, the logic circuit, the encoder 20, the decoder 30, the processor 43, the memory 44, and / or the display device 45 can communicate with each other. As described, although the video coding system 10B is depicted by using the encoder 20 and the decoder 30, the video coding system 10B may include only the encoder 20 or only the decoder 30 in different examples.

[0116] In some examples, the antenna 42 may be configured to transmit or receive encoded data of video data. Further, in some examples, the display device 45 may be configured to present the video data. In some examples, the logic circuit may be implemented by using theprocessing unit 46. The processing unit 46 may include ASIC logic, a graphics processor, a general -purpose processor, or the like. The video coding system 10B may alternatively include the optional processor 43. The optional processor 43 may similarly include ASIC logic, a graphics processor, a general-purpose processor, or the like. In some examples, the logic circuit may be implemented by hardware, for example, video coding dedicated hardware, and the processor 43 may be implemented by general -purpose software, an operating system, or the like. In addition, the memory 44 may be any type of memory, for example, a volatile memory (for example, a static RAM (SRAM) or a dynamic RAM (DRAM)) or a non-volatile memory (for example, a flash memory). In a non-limitative example, the memory 44 may be implemented by using a cache memory. In some examples, the logic circuit may access the memory 44 (for example, for implementation of an image buffer). In other examples, the logic circuit and / or the processing unit 46 may include a memory (for example, a cache) for implementation of an image buffer or the like.

[0117] In some examples, the encoder 20 implemented by using the logic circuit may include an image buffer (for example, implemented by using the processing unit 46 or the memory 44) and a graphics processing unit (for example, implemented by using the processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the encoder 20 implemented by using the logic circuit, to implement various modules that are described with reference to FIG. 2 and / or any other encoder system or subsystem described in this specification. The logic circuit may be configured to perform various operations described in this specification.

[0118] In some examples, the decoder 30 may be implemented by using the logic circuit in a similar manner, to implement various modules that are described with reference to the decoder 30 in FIG. 3 and / or any other decoder system or subsystem described in this specification. In some examples, the decoder 30 implemented by using the logic circuit may include an image buffer (implemented by using the processing unit 46 or the memory 44) and a graphics processing unit (for example, implemented by using the processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the decoder 30 implemented by using the logic circuit, to implement various modules that are described with reference to FIG. 3 and / or any other decoder system or subsystem described in this specification.

[0119] In some examples, the antenna 42 may be configured to receive encoded data of video data. As described, the encoded data may include data, an indicator, an index value, mode selection data, or the like related to video frame encoding described in this specification, forexample, data related to coding partitioning (for example, a transform coefficient or a quantized transform coefficient, an optional indicator (as described), and / or data defining coding partitioning). The video coding system 10B may further include the decoder 30 that is coupled to the antenna 42 and that is configured to decode the encoded data. The display device 45 is configured to present a video frame.

[0120] It is understood that in some embodiments of the present disclosure, for the example described with reference to the encoder 20, the decoder 30 may be configured to perform a reverse process. With regard to a signaling syntax element, the decoder 30 may be configured to receive and parse such a syntax element and correspondingly decode related video data. In some examples, the encoder 20 may entropy encode the syntax element into encoded video encoded data. In such examples, the decoder 30 may parse such a syntax element and correspondingly decode related video data.

[0121] FIG. 2 is a schematic / conceptual block diagram of an example of an encoder 20 configured to implement one or more embodiments of the present disclosure. In the example of FIG. 2, the encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (DPB) 230, a prediction processing unit 260, and an entropy encoding unit 270. The prediction processing unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a mode selection unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit. The encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder based on a hybrid video codec.

[0122] The encoder 20 receives, for example, via an input 202, a picture 201 or an image block 203 of the picture 201, for example, a picture in a sequence of pictures forming a video or a video sequence. The image block 203 may also be referred to as a current encoding block or a to-be-processed image block, and the picture 201 may be referred to as a current picture or a to-be-encoded picture (especially when the current picture is distinguished from another picture in video encoding, for example, the other picture is a previously encoded and / or decoded picture in a same video sequence, that is, a video sequence including the current picture).

[0123] The residual calculation unit 204 is configured to calculate a residual block 205 based on the picture image block 203 and a prediction block 265 (further details about the prediction block 265 are provided below), for example, obtain the residual block 205 in a sample domain by subtracting sample values of the prediction block 265 from sample values of the picture image block 203 sample by sample (pixel by pixel).

[0124] The transform processing unit 206 is configured to apply a transform, for example, a discrete cosine transform (DCT) or a discrete sine transform (DST), to sample values of the residual block 205 to obtain transform coefficients 207 in a transform domain. The transform coefficients 207 may also be referred to as transform residual coefficients and represent the residual block 205 in the transform domain.

[0125] The quantization unit 208 is configured to quantize the transform coefficients 207 to obtain quantized transform coefficients 209, for example, through scalar quantization or vector quantization.

[0126] The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 to quantized coefficients to obtain dequantized coefficients 211, for example, apply, based on or by using a same quantization step size as the quantization unit 208, the inverse of a quantization scheme applied by the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, and correspond, although usually different from the transform coefficients due to a loss caused by quantization, to the transform coefficients 207.

[0127] The inverse transform processing unit 212 is configured to apply the inverse transform of the transform applied by the transform processing unit 206, for example, an inverse DCT or an inverse DST, to obtain an inverse transform block 213 in the sample domain. The inverse transform block 213 may also be referred to as an inverse transform dequantized block 213 or an inverse transform residual block 213.

[0128] The reconstruction unit 214 (for example, a summer 214) is configured to add the inverse transform block 213 (that is, a reconstructed residual block 213) to the prediction block 265 to obtain a reconstructed block 215 in the sample domain, for example, by adding sample values of the reconstructed residual block 213 and the sample values of the prediction block 265.

[0129] For example, a buffer unit 216 (or a “buffer” 216), for example, a line buffer 216, is configured to buffer or store the reconstructed block 215 and a corresponding sample value, for example, for intra prediction. In other embodiments, the encoder may be configured to use unfiltered reconstructed blocks and / or corresponding sample values stored in the buffer unit 216 for any type of estimation and / or prediction, for example, intra prediction.

[0130] The loop filter unit 220 (or “loop filter” 220) is configured to filter the reconstructed block 215 to obtain the filtered block 221, to smooth pixel transition or improve video quality. The DPB 230 may be a reference picture memory that stores reference picture data for use in video data encoding by the encoder 20. The DPB 230 may be formed by any one of a varietyof memory devices, such as a DRAM (including a synchronous DRAM (SDRAM), a magnetoresistive RAM (MRAM), and a resistive RAM (RRAM)), or another type of memory devices. The DPB 230 and the buffer 216 may be provided by a same memory device or separate memory devices. In an example, the DPB 230 is configured to store the filtered block 221. The DPB 230 may be further configured to store other previously filtered blocks, for example, previously reconstructed and filtered blocks 221, of the same current picture or of different pictures, for example, previously reconstructed pictures, and may provide complete previously reconstructed, namely, decoded, pictures (and corresponding reference blocks and samples) and / or a partially reconstructed current picture (and corresponding reference blocks and samples), for example, for inter prediction. In an example, if the reconstructed block 215 is reconstructed without in-loop filtering, the DPB 230 is configured to store the reconstructed block 215.

[0131] The prediction processing unit 260, also referred to as a block prediction processing unit 260, is configured to receive or obtain the image block 203 (a current image block 203 of the current picture 201) and reconstructed picture data, for example, reference samples of the same (current) picture from the buffer 216 and / or reference picture data 231 of one or more previously decoded pictures from the DPB 230, and to process such data for prediction, namely, to provide the prediction block 265 that may be an inter prediction block 245 or an intra prediction block 255.

[0132] The mode selection unit 262 may be configured to select a prediction mode (for example, an intra or inter prediction mode) and / or a corresponding prediction block 245 or 255 to be used as the prediction block 265, for calculation of the residual block 205 and for reconstruction of the reconstructed block 215.

[0133] The inter prediction unit 244 may include a motion estimation (ME) unit (not shown in FIG. 2) and a motion compensation (MC) unit. The motion estimation unit is configured to receive or obtain the picture image block 203 (the current picture image block 203 of the current picture 201) and the decoded picture 231, or at least one or more previously reconstructed blocks, such as one or more reconstructed blocks of other / different previously decoded pictures 231, to perform motion estimation based on the determined inter prediction mode. For example, a video sequence may include the current picture and the previously decoded pictures 231, or in other words, the current picture and the previously decoded pictures 231 may be a part of or form a sequence of pictures forming a video sequence.

[0134] The intra prediction unit 254 is configured to obtain, for example, receive, the picture block 203 (the current picture block) and one or more previously reconstructed blocks, forexample, reconstructed neighbor blocks, of the same picture for intra estimation. The encoder 20 may be, for example configured to select an intra prediction mode from a plurality of (predetermined) intra prediction modes.

[0135] The entropy encoding unit 270 is configured to apply an entropy encoding algorithm or scheme (for example, a variable length coding (VLC) scheme, a context adaptive VLC (CAVLC) scheme, an arithmetic coding scheme, context adaptive binary arithmetic coding (CABAC), syntax-based CABAC (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method or technology) to a single one or all (or none) of the quantized residual coefficients 209, the inter prediction parameter, the intra prediction parameter, and / or the loop filter parameter to obtain encoded picture data 21 that can be output by an output 272, for example, in a form of an encoded data 21. The encoded data may be transmitted to the decoder 30 or archived for later transmission or retrieval by the decoder 30. The entropy encoding unit 270 may be further configured to entropy encode another syntax element for a current video slice being encoded.

[0136] FIG. 3 is a schematic / conceptual block diagram of an example of a decoder 30 configured to implement one or more embodiments of the present disclosure. The decoder 30 is configured to receive, for example, encoded picture data (for example, the encoded data) 21 encoded by the encoder 20 to obtain a decoded picture 331. During decoding, the decoder 30 receives video data from the encoder 20, for example, an encoded video encoded data representing a picture block of an encoded video slice and an associated syntax element.

[0137] In the example of FIG. 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (for example, a summer 314), a buffer 316, a loop filter 320, a DPB 330, and a prediction processing unit 360. The prediction processing unit 360 may include an inter prediction unit 344, an intra prediction unit 354, and a mode selection unit 362. In some examples, the decoder 30 may perform a decoding process substantially inverse to an encoding process performed by the encoder 20 described above with reference to FIG. 2.

[0138] The entropy decoding unit 304 is configured to perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and / or decoded coding parameters, for example, any one or all of an inter prediction parameter, an intra prediction parameter, a loop filter parameter, and / or another syntax element (decoded). The entropy decoding unit 304 is further configured to forward the inter prediction parameter, the intra prediction parameter, and / or the other syntax element to the prediction processing unit360. The decoder 30 may receive a syntax element at a video slice level and / or a video block level.

[0139] The inverse quantization unit 310 may have a same function as the inverse quantization unit 210, the inverse transform processing unit 312 may have a same function as the inverse transform processing unit 212, the reconstruction unit 314 may have a same function as the reconstruction unit 214, the buffer 316 may have a same function as the buffer 216, the loop filter 320 may have a same function as the loop filter 220, and the DPB 330 may have a same function as the DPB 230.

[0140] The prediction processing unit 360 may include an inter prediction unit 344 and an intra prediction unit 354. The inter prediction unit 344 may resemble the inter prediction unit 244 in function, and the intra prediction unit 354 may resemble the intra prediction unit 254 in function. The prediction processing unit 360 is usually configured to perform block prediction and / or obtain a prediction block 365 from the encoded data 21, and (explicitly or implicitly) receive or obtain a prediction-related parameter and / or information about a selected prediction mode, for example, from the entropy decoding unit 304.

[0141] The prediction processing unit 360 is configured to determine prediction information for a video block of the current video slice by parsing the motion vector and the other syntax element, and use the prediction information to generate the prediction block for the current video block being decoded. In one example of the present disclosure, the prediction processing unit 360 uses some of the received syntax elements to determine a prediction mode (for example, intra or inter prediction) for encoding video blocks of the video slice, an inter prediction slice type (for example, a B slice, a P slice, or a generalized P / B (GPB) slice), construction information for one or more of the reference picture lists for the slice, a motion vector for each inter-encoded video block of the slice, an inter prediction status for each interencoded video block of the slice, and other information, to decode the video blocks in the current video slice. In another example of this disclosure, a syntax element received by the decoder 30 from encoded data includes a syntax element in one or more of an adaptive parameter set (APS), a sequence parameter set (SPS), a picture parameter set (PPS), or a slice header.

[0142] The inverse quantization unit 310 may be configured to inversely quantize (namely, dequantize) quantized transform coefficients provided in the encoded data and decoded by the entropy decoding unit 304. An inverse quantization process may include using a quantization parameter calculated by the encoder 20 for each video block in the video slice, to determine adegree of quantization to be applied and also determine a degree of inverse quantization to be applied.

[0143] The inverse transform processing unit 312 is configured to apply an inverse transform (for example, an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to transform coefficients to generate residual blocks in a pixel domain.

[0144] The reconstruction unit 314 (for example, the summer 314) is configured to add an inverse transform block 313 (that is, a reconstructed residual block 313) to the prediction block 365 to obtain a reconstructed block 315 in a sample domain, for example, by adding sample values of the reconstructed residual block 313 and sample values of the prediction block 365.

[0145] The loop filter unit 320 (in a coding loop or after the coding loop) is configured to filter the reconstructed block 315 to obtain a filtered block 321 , to smooth pixel transition or improve video quality. In an example, the loop filter unit 320 may be configured to perform any combination of filtering technologies described below. The loop filter unit 320 is intended to represent one or more loop filters including a de-blocking filter, a SAO filter, and another filter, for example, a bilateral filter, an ALF, a sharpening or smoothing filter, or a collaborative filter. Although the loop filter unit 320 is shown in FIG. 3 as an in-loop filter, the loop filter unit 320 may be implemented as a post-loop filter in other configurations.

[0146] The decoded video blocks 321 in a given frame or picture are then stored in the DPB 330 that stores reference pictures used for subsequent motion compensation.

[0147] The decoder 30 is configured to, for example, output the decoded picture 31 by using an output 332, for presentation to a user or viewing by a user.

[0148] It is understood that, in the encoder 20 and the decoder 30 in the present disclosure, a processing result of a specific phase may be further processed and then output to the next phase. For example, after interpolation filtering, motion vector derivation or loop filtering, a further operation, such as clip or shift, may be performed on the processing result of a corresponding phase.

[0149] As mentioned above, the transform used in video coding may include DCT or DST. The transform used in video coding may further include wavelet transform. The wavelet transform is a localized transform that may perform localized and multi-scale analysis on images, focusing on detailed changes of image signals. It is suitable for image coding tasks.

[0150] A video codec suitable for wavelet transform may be provided according to embodiments of the present disclosure, which may independently perform video coding on low-frequency subbands and high-frequency subbands, effectively reducing video coding complexity and improving video coding efficiency. For example, an image is decomposed intoa low-frequency subband and high-frequency subbands through the wavelet transform, and the low-frequency subband and the high-frequency subbands are encoded to generate a low-frequency subband bitstream and a high-frequency subband bitstream respectively. The low-frequency subband may be understood as a subimage representing a low-frequency signal (or low-frequency information) of an original image, and the high-frequency subbands may be understood as subimages representing high-frequency signals (or high-frequency information) of the original image. It is noted that the subband may be further performed with the wavelet transform to obtain a low-frequency subband and high-frequency subbands. That is, the wavelet transform is performed recursively.

[0151] FIG. 4 is a schematic / conceptual block diagram of an example of an encoder 40 configured to implement one or more embodiments of the present disclosure. In an example of FIG. 4, the encoder 40 may include, but is not limited to, a subfigure partitioning unit 410, a wavelet forward transform unit 420, a low-frequency subband processing path 430, and a high-frequency subband processing path 440.

[0152] The subfigure partitioning unit 410 is used to obtain a current figure and partition the current figure to obtain at least one subfigure. The subfigure partitioning unit 410 partitions the current figure into N subfigures in a subfigure partitioning manner, where N is an integer greater than 0 (or greater than 1). The subfigure partitioning manner may include, but is not limited to, at least one of the following:

[0153] a width and / or height of the subfigure being a multiple of 128;

[0154] a maximum width of the subfigure being 1024 pixels;

[0155] a minimum height and / or width of the subfigure being 256 pixels;

[0156] a resolution of the original figure being less than or equal to 1080p, where N is an integer greater than 1 and less than or equal to 8; or

[0157] a length of the original figure being greater than or equal to 4320 pixels, and a width of the original figure being greater than or equal to 2160 pixels, where N is an integer greater than 1 and less than or equal to 16; or

[0158] the length of the original figure being greater than or equal to 7680 pixels, and the width of the original figure being greater than or equal to 4320 pixels, where N is an integer greater than 1 and less than or equal to 32.

[0159] The above numerical values are for illustrative purposes only and may be set according to actual needs.

[0160] FIG. 5 is a schematic diagram of an example of subfigure partitioning. In an example of FIG. 5, the subfigure partitioning unit 410 partitions a figure 401 into m*n subfigures in thesubfigure partitioning manner. For example, in this example, a width and height of each subfigure are multiples of 128.

[0161] The subfigure partitioning unit 410 partitions the figure into m*n subfigures in the subfigure partitioning manner. For sub figures whose widths and heights are not multiples of 128, the subfigure partitioning unit 410 may pad these subfigures so that heights and widths of the padded subfigures may be multiples of 16.

[0162] The wavelet forward transform unit 420 is used to perform the wavelet transform (which also may be referred to as wavelet forward transform) on the subfigure to obtain a low-frequency subband and high-frequency subbands. The low-frequency subband includes a low-frequency signal in the subfigure, that satisfy low-frequency filter coefficients, and the high-frequency subbands include high-frequency signals in the subfigure, decomposed by a high-frequency filter in the wavelet transform.

[0163] FIG. 6 is a schematic diagram of an example of a wavelet transform for implementing one or more embodiments of the present disclosure. In an example of FIG. 6, the wavelet forward transform unit 420 obtains the current subfigure, such as subfigure 1 1. The wavelet forward transform unit 420 performs a wavelet transform on the current subfigure. The wavelet transform includes one horizontal wavelet transform and one vertical wavelet transform, so as to obtain wavelet coefficients of a low-low (LL) subband (referred to simply as an LL subband), wavelet coefficients of a low-high (LH) subband (referred to simply as an LH subband), wavelet coefficients of a high-high (HH) subband (referred to simply as an HH subband), and wavelet coefficients of a high-low (HL) subband (referred to simply as an HL subband).

[0164] In embodiments of the present disclosure, the low-frequency subband includes the LL subband, and the high-frequency subbands include the LH subband, the HH subband, and the HL subband. For example, the LL subband, LH subband, HH subband, and HL subband have a same size (including width and height).

[0165] Referring back to FIG. 4, the low-frequency subband processing path 430 is used to obtain the wavelet coefficients 403 of the low-frequency subband to obtain low-frequency subband encoded data 413. The low-frequency subband processing path 430 includes, but is not limited to, a block partitioning unit 431 (which also may be referred to as a low-frequency subband block partitioning unit 431), a residual calculation unit 432, a prediction unit 437, a control unit 438, a transform / quantization unit 433 (which also may be referred to as a low-frequency subband transform / quantization unit 433), an inverse quantization / inverse transform unit 434 (which also may be referred to as a low-frequency subband inversequantization / inverse transform unit 434), a low-frequency subband reconstruction unit 435, a low-frequency subband splicing unit 436, a low-frequency subband entropy encoding unit 439.

[0166] The high-frequency subband processing path 440 is used to obtain the high-frequency subbands to obtain high-frequency subband encoded data. The high-frequency subband processing path 440 includes, but is not limited to, a block partitioning unit 431 (which also may be referred to as a high-frequency subband block partitioning unit 431 or a second block partitioning unit 431), a transform / quantization unit 442 (which also may be referred to as a high-frequency subband transform / quantization unit 442, or a second transform / quantization unit 442), and a high-frequency subband entropy encoding unit 443.

[0167] For example, in some instances, the encoder 40 may include more or fewer units or modules than those shown in FIG. 4.

[0168] The following describes an encoding method provided in the present disclosure in detail with reference to the encoder shown in FIG 4.

[0169] The codec receives the figure 401. The description of the figure 401 may be found above and will not be repeated here.

[0170] The subfigure partitioning unit 410 partitions the figure 401 into N subfigures and outputs the partitioned N subfigures, where N is an integer greater than 0. In embodiments of the present disclosure, each subfigure is independently encoded and decoded. During the encoding process, the subfigure 402 may be called the current subfigure or the subfigure to be encoded.

[0171] The wavelet forward transform unit 420 performs a wavelet forward transform on the current subfigure to obtain wavelet coefficients 403 of a low-frequency subband (referred to simply as a low-frequency subband) and wavelet coefficients 414 of high-frequency subbands (referred to simply as the high-frequency subbands) of the current subfigure. The wavelet coefficients 403 of the low-frequency subband include the wavelet coefficients of the LL subband, and the high-frequency subbands include the wavelet coefficients of the LH subband, the wavelet coefficients of the HL subband, and the wavelet coefficients of the HH subband. In embodiments of the present disclosure, each subfigure of the figure may be independently encoded and decoded, and the high-frequency subbands and the low-frequency subband of each subfigure are independently encoded and decoded. The LH subband, the HL subband, and the HH subband in the high-frequency subbands may be independently encoded and decoded.

[0172] The block partitioning unit 431 (which may be referred to the low-frequency subband block partitioning unit) is used to obtain the wavelet coefficients 403 of the low-frequency subband of the current subfigure 402 to obtain at least one macroblock 404 of the low-frequency subband of the subfigure 402. The macroblock may be understood as a collection of partial coefficients in the wavelet coefficients of the low-frequency subband.

[0173] The block partitioning unit 431 partitions the wavelet coefficients 403 of the low-frequency subband of the current subfigure 402 into at least one macroblock 404 of the low-frequency subband of the current subfigure in a block partitioning manner, for example, M macroblocks, where M is an integer greater than 0 (or greater than 1). The low-frequency block partitioning unit 431 outputs the macroblocks 404 of the wavelet coefficients 403 of the current low-frequency subband one by one to the residual calculation unit 432 and the control unit 438.

[0174] In embodiments of the present disclosure, the macroblock 404 is a basic video coding unit. During the encoding process, the macroblock 404 may be referred to as a current block, a current figure block, a macroblock 404 to be encoded, a block to be encoded, a figure block to be encoded, etc.

[0175] For example, the block partitioning manner includes, but is not limited to the following.

[0176] The wavelet coefficients 403 of the low-frequency subband are partitioned into macroblocks 404 of 8 x 8 (in pixels) as a basic encoding unit.

[0177] For example, as mentioned above, each subband takes the macroblock 404 as the basic encoding unit. The current macroblock 404 to be encoded is referred to as the current macroblock 404. The low-frequency subband processing path 430 encodes each macroblock 404 of the wavelet coefficients 403 of the low-frequency subband block by block. For example, encoding and prediction are performed on each macroblock 404. The following only describes the encoding process of the current macroblock 404, and processing of other macroblocks is the same, which will not be repeated therein. For example, during encoding, it refers to a macroblock currently being encoded; and during decoding, it refers to a macroblock currently being decoded. A decoded macroblock in a reference figure used to predict the current macroblock 404 is referred to as a reference block (i.e., a low-frequency subband reconstruction block 409). That is, the reference block is a block that provides a reference signal for the current block. The reference signal represents pixel values in the macroblock 404. A block in the reference figure that provides a prediction signal for the current block is referred to as a prediction block 405. The prediction signal represents the pixel values, sample values, or sampling signals in the prediction block 405. For example, after traversing a plurality of reference blocks, an optimal reference block is found. This optimal reference block will provide prediction for the current block, and is referred to as the prediction block 405.

[0178] Referring back to FIG. 4, the residual calculation unit 432 is used to obtain the current macroblock 404 and the prediction block 405 (details of the prediction block 405 will beprovided below) to obtain a residual block 406. The residual calculation unit 432 performs residual calculation on the current macroblock 404 and the prediction block 405 to obtain the residual block 406. The residual calculation unit 432 outputs the residual block 406 to the transform / quantization unit 433.

[0179] The transform / quantization unit 433 is used to obtain the residual block 406, so as to obtain a residual quantization block 407. The transform / quantization unit 433 performs transform and / or quantization processing on the residual block 406 to obtain the residual quantization block 407, which may also be referred to as a quantized residual block. The residual quantization block 407 includes quantization coefficients of the residual block 406.

[0180] The transform / quantization unit 433 outputs the residual quantization block 407 to the inverse quantization / inverse transform unit 434 and the low-frequency subband entropy encoding unit 439.

[0181] The inverse quantization / inverse transform unit 434, also referred to as the dequantization / reverse transform unit, is used to obtain residual coefficients 407 to obtain a reconstructed residual block 408. The inverse quantization / inverse transform unit 434 performs inverse quantization and / or inverse transform processing on the residual quantization block 407 to obtain the reconstructed residual block 408, which may also be referred to as an inverse quantized residual block. The reconstructed residual block 408 includes inverse quantized coefficients of the residual block 406. The inverse quantization / inverse transform unit 434 outputs the reconstructed residual block 408 to the low-frequency subband splicing unit 436.

[0182] The inverse quantization / inverse transform unit 434 may include an inverse quantization unit and an inverse transform unit (not shown in FIG. 4). The inverse quantization unit is used to perform inverse quantization on the input coefficients, and the inverse transform unit is used to perform inverse transform on the input coefficients.

[0183] The low-frequency subband reconstruction unit 435 is used to obtain a low-frequency subband reconstruction block 409 based on the prediction block 405 and the reconstructed residual block 408. The low-frequency subband reconstruction unit 435 adds the reconstructed residual block 408 to the prediction block 405 to obtain the low-frequency subband reconstruction block 409, which may also be referred to as a reconstructed low-frequency subband macroblock. For example, the low-frequency subband reconstruction unit 435 outputs the low-frequency subband reconstruction block 409 to the prediction unit 437 and the low-frequency subband splicing unit 436. For example, the low-frequency subband reconstruction unit 435 outputs the low-frequency subband reconstruction block 409 to the control unit 438.

[0184] The low-frequency subband splicing unit 436 is used to obtain a reconstructed low-frequency subband 411 based on the low-frequency subband reconstruction block 409, which may also be referred to as reconstructed values or reconstructed data of the low-frequency subband. For example, the low-frequency subband splicing unit 436 outputs the reconstructed low-frequency subband 411 to the prediction unit 437. For example, the low-frequency subband splicing unit 436 outputs the reconstructed low-frequency subband 411 to the control unit 438.

[0185] As mentioned above, the low-frequency subband takes the macroblock as the basic encoding unit. The low-frequency subband splicing unit 436 may obtain M low-frequency subband reconstruction blocks of the low-frequency subband. The low-frequency subband splicing unit 436 may reconstruct a corresponding low-frequency subband based on the M low-frequency subband reconstruction blocks, that is, may obtain the reconstructed low-frequency subband 411.

[0186] The control unit 438 is used to obtain control information 418 and output the control information 418 to the low-frequency subband entropy encoding unit 439. For example, the control information 418 may include, but is not limited to, mode information 412. The control unit 438 may determine the mode information 412 based on the macroblock 404 (i.e., the current block), or may determine the mode information 412 based on the current macroblock 404, the low-frequency subband reconstruction block 409, and the reconstructed low-frequency subband 411.

[0187] The mode information may also be referred to as prediction mode information, etc. The mode information is used to indicate a prediction mode (or prediction manner) of the prediction unit 437, such as inter-frame or intra-frame prediction mode. The control unit 438 may output the mode information 412 to the prediction unit 437.

[0188] The prediction unit 437, also referred to as a prediction processing unit, is used to obtain the mode information 412 and perform prediction processing based on the mode information 412. The prediction unit 437 may select the prediction mode based on the mode information 412. In an example, the prediction unit 437 may obtain the low-frequency subband reconstruction block 409 based on the mode information 412, so as to obtain the prediction block 405. The prediction unit 437 may perform intra-frame prediction based on the low-frequency subband reconstruction block 409 to obtain the prediction block 405. In another example, the prediction unit 437 may obtain the reconstructed low-frequency subband 411 based on the mode information 412, so as to obtain the prediction block 405.

[0189] The prediction unit 437 outputs the prediction block 405 to the residual calculation unit 432 and the low-frequency subband splicing unit 436.

[0190] The low-frequency subband entropy encoding unit 439 is used to obtain the low-frequency subband encoded data 413 (which may also be referred to as an encoded low-frequency subband) based on the quantized residual block 407 and the mode information 412. The low-frequency subband entropy encoding unit 439 uses entropy encoding algorithms or schemes (e.g., VLC scheme, CAVLC scheme, arithmetic coding scheme, CABAC, SBAC, PIPE coding, or other entropy encoding methods or techniques) to perform entropy encoding on the quantized residual block 407 and the mode information 412 to obtain the low-frequency subband encoded data 413, which is output in a form of an encoded bitstream.

[0191] Referring again to FIG. 4, the block partitioning unit 441, also referred to as a high-frequency subband block partitioning unit, is used to obtain high-frequency subbands of the current subfigure 402 to obtain at least one macroblock 415 of the high-frequency subbands of the subfigure 402. The specific description may refer to the low-frequency subband part and will not be repeated here. The block partitioning unit 441 partitions the high-frequency subbands 414 of the current sub figure 402 (hereinafter referred to as current high-frequency subbands) into at least one macroblock 415 of the current subfigure 402 based on the block partitioning manner, for example, M macroblocks, where M is an integer greater than 0 (or greater than 1). Other parts not described can refer to the relevant description of the block partitioning unit 431 and will not be repeated herein.

[0192] The block partitioning unit 441 outputs the macroblocks of the current high-frequency subbands one by one to the quantization / transform unit 442.

[0193] The transform / quantization unit 442 is used to perform transform and / or quantization processing on the macroblock 415 to obtain quantized coefficients 416 of the high-frequency subband block (i.e., quantized coefficients of the current macroblock). The transform / quantization unit 442 outputs the quantized coefficients 416 of the high-frequency subband block to the high-frequency subband entropy encoding unit 443.

[0194] The high-frequency subband entropy encoding unit 443 is used to perform entropy encoding on the data to be encoded to obtain the high-frequency subband encoded data 417. The data to be encoded may include, but is not limited to, quantized coefficients and syntax elements of each high-frequency subband block in the high-frequency subbands. The high-frequency subband encoded data 417 includes, but is not limited to, HH subband encoded data, HL subband encoded data, LH subband encoded data.

[0195] The high-frequency subband entropy encoding unit 443 uses entropy encoding algorithms or schemes to encode the data to be encoded. The entropy encoding schemes may include, for example, at least one encoding scheme of VLC scheme, CAVLC scheme, arithmetic coding scheme, or CABAC scheme.

[0196] The video coding method provided in the present disclosure supports both full I-frame configuration and I / P frame alternate encoding configuration. An architecture of the encoder shown in FIG. 4 adds relevant modules required for the prediction process on the basis of the wavelet transform architecture, which may improve compression efficiency of I / P frame alternate encoding for scenarios such as fixed camera positions and slow camera movements.

[0197] For example, the encoder 40 may further include, but is not limited to, a combination unit (not shown in FIG. 4), which may also be called a multiplexer (MUX), etc. The combination unit is used to generate a figure bitstream based on the low-frequency subband encoded data and the high-frequency subband encoded data.

[0198] The combination unit writes the low-frequency subband encoded data 413 into the figure bitstream, and writes the high-frequency subband encoded data 417 into the figure bitstream. In embodiments of the present disclosure, the low-frequency subband and the high-frequency subbands are encoded separately, so that the low-frequency subband encoded data and the high-frequency subband encoded data may be independently decoded. That is, at a decoding side, it may independently decode the low-frequency subband encoded data and the high-frequency subband encoded data in the figure bitstream, thereby improving decoding efficiency.

[0199] FIG. 7A is a schematic / conceptual block diagram of an example of a decoder for implementing one or more embodiments of the present disclosure. In an example of FIG. 7 A, for example, the decoder receives the figure bitstream encoded by the encoder to obtain a decoding figure of the original figure, which may also be referred to as a decoded figure, a reconstructed figure, etc. During a decoding process, the decoder receives the figure bitstream from the encoder, which may include, but is not limited to, figure header information and figure data.

[0200] In the example shown in FIG. 7A, the decoder includes, but is not limited to, a low-frequency subband processing path 710, a high-frequency subband processing path 720, a wavelet inverse transform unit 730, a figure combination unit 740 (which may also be referred to as a figure splicing unit).

[0201] For example, the low-frequency subband processing path 710 is used to obtain low-frequency subband encoded data in order to obtain a reconstructed low-frequency subband 706(which may also be referred to as a decoded low-frequency subband). The low-frequency subband processing path 710 includes, but is not limited to, a low-frequency subband entropy decoding unit 711, an inverse quantization / inverse transformation unit 712 (which may also be referred to as a low-frequency subband inverse quantization / inverse transformation unit), a low-frequency subband reconstruction unit 713, a low-frequency subband splicing unit 715, a prediction unit 714.

[0202] The high-frequency subband processing path 720 is used to obtain high-frequency subband encoded data to obtain a reconstructed high-frequency subband 731, which may also be referred to as reconstructed values of the high-frequency subbands or reconstructed data of the high-frequency subbands, etc., including but not limited to, a high-frequency subband entropy decoding unit 721, an inverse quantization / inverse transformation unit 722 (which may also be referred to as a high-frequency subband inverse quantization / inverse transformation unit), a high-frequency subband reconstruction unit 723.

[0203] In some instances, the decoder shown in FIG. 7A may perform a decoding process substantially inverse to the encoding process performed by the encoder described with reference to FIG. 4.

[0204] The following describes the decoding method in the embodiments of the present disclosure in detail with reference to the decoder 70 shown in FIG. 7A.

[0205] For example, the decoder 70 may obtain the high-frequency subband encoded data and the low-frequency subband encoded data in the figure bitstream based on figure header information in the figure bitstream. Furthermore, as described above, the encoder uses the macroblock as the basic encoding unit during the encoding process, and accordingly, the decoder also uses the macroblock (for example, the encoded macroblock) as the basic decoding unit for decoding during the decoding process.

[0206] For example, the low-frequency subband entropy decoding unit 711 performs entropy decoding on the low-frequency subband encoded data 701 in the figure bitstream, and uses the macroblock as the basic decoding unit to obtain syntax elements and quantized coefficients 702 of the low-frequency subband block (i.e., quantized coefficients of the current macroblock). The description of the quantized coefficients 702 of the low-frequency subband may refer to the encoder side, which is not repeated herein. The low-frequency subband entropy decoding unit 711 obtains the encoded macroblocks of the low-frequency subband (for example, the LL subband) in each subfigure in the figure bitstream (i.e., the encoded data of the macroblocks), and performs entropy decoding on each encoded macroblock to obtain the quantized coefficients 702 and syntax elements of the low-frequency subband corresponding to eachencoded macroblock (which may be simply referred to as the quantized coefficients of the macroblock of the low-frequency subband). During the decoding process, the encoded macroblock that is currently decoded may be referred to as the current block.

[0207] The low-frequency subband entropy decoding unit 711 is used to output the quantized coefficients 702 of the low-frequency subband blocks to the inverse quantization / inverse transformation unit 712, and to output the mode information 412 to the prediction unit 714.

[0208] The inverse quantization / inverse transformation unit 712 is used to obtain the quantized coefficients 702 of the low-frequency subband blocks, in order to obtain the inverse quantized coefficients 703 of the low-frequency subband blocks. It may also be the inverse transformation coefficients of the current block of the low-frequency subband (depending on whether inverse transformation processing has been performed). The inverse quantization / inverse transformation unit 712 performs inverse quantization, or inverse quantization and inverse transformation on the quantized coefficients of the current block of the low-frequency subband, to obtain the inverse quantization coefficients of the current block of the low-frequency subband. The inverse quantization / inverse transformation unit 712 outputs the inverse quantization coefficients 703 of the low-frequency subband blocks (e.g., the inverse quantization coefficients of the current block of the low-frequency subband) to the low-frequency subband reconstruction unit 713.

[0209] The low-frequency subband reconstruction unit 713 is used to obtain the low-frequency subband reconstruction block 704 based on the quantization coefficients 703 of the low-frequency subband and the prediction block 705, which may also be referred to as reconstruction coefficients of the low-frequency subband block. The low-frequency subband reconstruction unit 713 adds a prediction block to the quantization coefficients of the current block of the low-frequency subband to obtain the low-frequency subband reconstruction block 704 corresponding to the current macroblock.

[0210] The prediction unit 714 is used to obtain the mode information 412 and performs corresponding prediction processing according to the mode information 412. For example, intra-frame prediction may be performed based on the low-frequency subband reconstruction block 704, or inter-frame prediction may be performed based on the reconstructed low-frequency subband 706. The execution manner may refer to the encoding side, which is not repeated herein. The prediction unit 714 outputs the prediction block 705 to the low-frequency subband reconstruction block 704.

[0211] For example, the high-frequency subband entropy decoding unit 721 obtains the high-frequency subband encoded data 707 in the figure bitstream, and uses the macroblock as thebasic decoding unit to obtain quantized coefficients 708 of each high-frequency subband block in the high-frequency subband (as the quantized coefficients of the current macroblock). The high-frequency subband entropy decoding unit 721 performs entropy decoding on the current block of the high-frequency subband encoded data 707 to obtain the quantized coefficients of the current block of the high-frequency subband. For example, control information (such as including mode information 412) corresponding to the current block may further be obtained based on entropy decoding. The high-frequency subband entropy decoding unit 721 outputs the quantized coefficients 708 of the high-frequency subband block to the inverse quantization / inverse transformation unit 722.

[0212] The inverse quantization / inverse transformation unit 722, also called the high-frequency subband inverse quantization / inverse transformation unit, is used to obtain the quantized coefficients 708 of the high-frequency subband blocks to obtain the reconstruction coefficients 709 of the high-frequency subband blocks. The reconstruction coefficients may be dequantized coefficients after dequantization processing, or inverse transformation coefficients after dequantization and inverse transformation processing.

[0213] The high-frequency subband reconstruction unit 723 (also referred to as the high-frequency subband splicing unit) is used to obtain the reconstruction coefficients 709 of the high-frequency subband blocks to obtain reconstructed high-frequency subbands 731, which may also be referred to as reconstructed values of the high-frequency subbands or reconstructed data of the high-frequency subbands, etc. The high-frequency subband reconstruction unit 723 may obtain reconstruction coefficients corresponding to each macroblock of the high-frequency subbands, i.e., the reconstructed high-frequency subband block. The high-frequency subband reconstruction unit 723 may splice the obtained macroblocks to obtain corresponding high-frequency subbands. The reconstructed high-frequency subbands for example include a reconstructed HL subband (for example, reconstruction coefficients of the HL subband), a reconstructed HH subband (for example, reconstruction coefficients of the HH subband), a reconstructed LH subband (for example, reconstruction coefficients of the LH subband).

[0214] The wavelet inverse transform unit 730 is used to obtain the reconstructed high-frequency subbands 731 and the reconstructed low-frequency subband 706, in order to obtain the reconstructed subfigure 732. The wavelet inverse transform unit 730 obtains the reconstructed low-frequency subband 706 output by the low-frequency subband splicing unit 715, as well as the reconstructed high-frequency subbands 731 output by the high-frequency subband reconstruction unit 723, and performs wavelet inverse transformation on the reconstructed low-frequency subband 706 and the reconstructed high-frequency subbands 731to obtain the reconstructed subfigure 732. The wavelet inverse transform unit 730 outputs the reconstructed subfigure 732 to the figure composition unit (which can also be referred to as the figure splicing unit, etc.).

[0215] The figure composition unit 740 is used to obtain the reconstructed subfigure 732, in order to obtain the reconstructed figure 733 of the original figure, which may also be referred to as the decoding figure or the decoded figure, etc. The figure composition unit 740 may obtain N reconstructed subfigures (N being an integer greater than 0) of the figure (referring to the original figure), and splice (or combine) the N reconstructed subfigures in the partitioning manner (including size and position) of each reconstructed subfigure during encoding, in order to obtain the reconstructed figure 733.

[0216] For example, after obtaining the reconstructed figure, the figure composition unit 740 may determine whether the reconstructed figure includes padding based on the figure size information in the figure header information and the size information of the current reconstructed figure. In one example, if the size of the current reconstructed figure is the same as a size indicated by the figure size information (i.e., the same as a size of the original figure), the figure composition unit 740 may send the reconstructed figure to the display device. In this case, sizes of the displayed figure, the original figure, and the reconstructed figure are the same. In another example, if the size of the current reconstructed figure is different from the size indicated by the figure size information (e.g., larger than the size of the original figure), the figure composition unit 740 may remove the padding parts of the current reconstructed figure based on the size indicated by the figure size information to obtain the displayed figure. The size of the displayed figure is the same as the size of the original figure. For example, the decoder is used to, for example, output the reconstructed figure via an output port (or output interface) of the decoder for presentation to the user or for viewing by the user.

[0217] Other variations of the decoder may be used to decode the compressed figure bitstream.

[0218] FIG. 7B is a schematic / conceptual block diagram of an example decoder for implementing one or more embodiments of the present disclosure. In this example, the inverse wavelet transform unit 730 is used to obtain the reconstructed coefficients 709 of the high-frequency subband block (i.e., the high-frequency subband reconstruction block) and the low-frequency subband reconstruction block 704 output by the low-frequency subband reconstruction block unit 713, and performs an inverse wavelet transform on the high-frequency subband reconstruction block (e.g., including the HH subband reconstructed block, HL subband reconstructed block, LH subband reconstructed block) and the low-frequency subband reconstruction block 704 to obtain the reconstructed block 734, which is thereconstructed block of the current subfigure and may also be referred to as the reconstructed data or reconstructed values of the current block of the current subfigure.

[0219] The inverse wavelet transform unit 730 outputs the reconstructed block 734 to the subfigure composition unit 750. The subfigure composition unit 750 may obtain the reconstructed subfigure 732 of the current subfigure based on at least one reconstructed block corresponding to the current subfigure, which may also be referred to as the reconstructed values or reconstructed data of the current subfigure, etc. The undescribed parts of FIG. 7B may refer to FIG. 7A, and will not be repeated herein.

[0220] The bitstream output by the encoder in the embodiments of the present disclosure will be described in detail below. A bitstream structure described below may be applied to the encoder shown in FIG. 4, and of course, it may also be applied to other variations of the encoder based on FIG. 4.

[0221] FIG. 8 is a schematic diagram of a structure of an example of a figure bitstream. In an example of FIG. 8, the figure bitstream includes, but is not limited to, figure header information and figure data (which may also be referred to as figure data region(s)).

[0222] For example, the figure data includes at least one figure data region (which may also be referred to as figure data subregion). For example, the at least one figure data region includes, but not limited to, a first figure data region and a second figure data region. During the encoding process, the encoder (e.g., via the composition unit) writes the high-frequency subband encoded data and the low-frequency subband encoded data into the figure bitstream. The encoder writes the high-frequency subband encoded data into the first figure data region and writes the low-frequency subband encoded data into the second figure data region. A writing order of the encoded data is only an illustrative example and will not be repeated below.

[0223] For example, the figure header information includes, but is not limited to, offset information and figure size information.

[0224] For example, the figure size information is used to indicate the size of the original figure. As described above, during the encoding process, some subfigures may be padded during the subfigure partitioning process to ensure that the length and width of each subfigure are multiples of 16. Thus, during decoding, the size of the reconstructed figure obtained by the decoder may be larger than the size of the original figure. The decoder may process the reconstructed figure based on the figure size information to remove the padding parts of the reconstructed figure.

[0225] For example, the offset information is used to indicate a position of the data region in the figure bitstream, which may also be understood as indicating a position of the independentlydecodable encoded data in the figure bitstream. When the encoded data (i.e., the figure bitstream) is decoded according to the present disclosure, the offset information in the figure header information may be used to obtain encoded data that may be independently encoded, and the decoding operation may be performed on the encoded data. The encoded data that may be independently decoded (e.g., the low-frequency subband encoded data and the high-frequency subband encoded data) may be decoded synchronously during decoding to improve decoding efficiency.

[0226] In one example, the offset information may be a length of a figure data region where adjacent encoded data that may be independently decoded in the figure bitstream is located.

[0227] In another example, the offset information may be an offset (i.e., difference) between a starting position of a figure data region where the encoded data that may be independently decoded is located and an end position of the figure header information.

[0228] It is noted that the embodiments of the present disclosure only take a figure bitstream of a single figure as an example. That is, for example, the bitstream includes one figure data. During the encoding process of video figures, the encoder may generate an encoded figure bitstream for each figure. That is, the figure bitstream includes multiple figure data, and each figure data carries encoded data of a corresponding figure.

[0229] In an example shown in FIG. 8, the high-frequency subband encoded data of each subfigure of the figure (including the LH subband encoded data, HH subband encoded data, and HL subband encoded data) are continuously written into the first figure data region.

[0230] For example, as shown in FIG. 8, a subfigure 1-HL-MB0, a subfigure 1-HH-MB0, and a subfigure 1-LH-MB0 are continuously written into the first figure data region. The subfigure 1-HL-MB0 represents an encoded macroblock MB0 of an HL subband of the sub figure 1, the subfigure 1-HH-MB0 represents an encoded macroblock MB0 of an HH subband of the subfigure 1, and the subfigure 1-LH-MB0 represents an encoded macroblock MB0 of an LH subband of the subfigure 1. The figure only shows an encoded data structure of the subfigure 1 in the bitstream, and other subfigures are similar, so they will not be described one by one.

[0231] In this example, during decoding, the decoder side may decode the high-frequency subband encoded data in an order of subfigures, that is, each subfigure in the first figure data region is independently decoded. The low-frequency subband encoded data is decoded in an order of subfigures, that is, each subfigure in the second figure data region is independently decoded. When decoding the first figure data region, the decoder side may decode each encoded macroblock in sequence according to an order of the encoded macroblocks of each subfigure in the region. That is, the three high-frequency subband encoded data of the subfigure1 are continuously written into the first figure data region. Therefore, during decoding, the three high-frequency subband encoded data of the subfigure 1 may be decoded one by one to obtain the decoded high-frequency subband. A structure of the LL subband encoded data also takes each subfigure as a granularity, and the encoded data of each subfigure includes the encoded data of the LL subband of each macroblock of the subfigure.

[0232] In the embodiments of the present disclosure, multiple encoded data that may be independently decoded may be decoded simultaneously, or one or more high-frequency subbands may be decoded simultaneously. The number of simultaneous decodings depends on hardware performance of the decoder.

[0233] FIG. 9 is a schematic diagram of a structure of a video coding device 900 (for example, a video encoding device 900 or a video decoding device 900) according to one or more embodiments of the present disclosure. The video coding device 900 is applicable to implementation of one or more embodiments described in this specification. In some embodiments, the video coding device 900 may be a video decoder (for example, the decoder 30 in FIG. 1A) or a video encoder (for example, the encoder 20 in FIG. 1A). In other some embodiments, the video coding device 900 may be one or more components of the decoder 30 in FIG. 1 A or the encoder 20 in FIG. 1 A.

[0234] The video coding device 900 includes an ingress port 910 and a receiver unit (Rx) 920 for receiving data, a processor, a logic unit, or a central processing unit (CPU) 930 for processing the data, a transmitter unit (Tx) 940 and an egress port 950 for transmitting the data, and a memory 960 for storing the data. The video coding device 900 may further include an optical-to-electrical conversion component and an electrical-to-optical (EO) component that are coupled to the ingress port 910, the receiver unit 920, the transmitter unit 940, and the egress port 950, for egress or ingress of optical or electrical signals.

[0235] The processor 930 is implemented by hardware and software. The processor 930 may be implemented as one or more CPU chips, cores (for example, multi -core processors), FPGAS, ASICs, and DSPs. The processor 930 communicates with the ingress port 910, the receiver unit 920, the transmitter unit 940, the egress port 950, and the memory 960. The processor 930 includes a coding module 970 (for example, an encoding module 970 or a decoding module 970). The encoding / decoding module 970 implements the embodiments disclosed in this specification, to implement methods provided in the embodiments of the present disclosure. For example, the encoding / decoding module 970 implements, processes, or provides various coding operations. Therefore, the encoding / decoding module 970 substantially improves functions of the video coding device 900 and affects transform of the video coding device 900to a different state. Alternatively, the encoding / decoding module 970 is implemented as instructions stored in the memory 960 and executed by the processor 930.

[0236] The memory 960 includes one or more disks, tape drives, and solid-state drives (SSDs) and may be used as an overflow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory 960 may be volatile and / or non-volatile, and may be a ROM, a RAM, a ternary content-addressable memory (TCAM), and / or a SRAM.

[0237] FIG. 10 is a simplified block diagram of a device 1000 that can be used as any one or two of the source device 12 and the destination device 14 in FIG. 1 A according to one or more example embodiments. The device 1000 may implement the technologies of the present disclosure. In other words, FIG. 10 is a schematic block diagram of an implementation of an encoding device or a decoding device (or a coding device 1000) according to one or more embodiments of the present disclosure. The coding device 1000 may include a processor 1010, a memory 1030, and a bus system 1050. The processor 1010 and the memory 1030 are connected through the bus system 1050. The memory 1030 is configured to store instructions. The processor 1010 is configured to execute the instructions stored in the memory 1030. The memory 1030 of the coding device 1000 stores program code, and the processor 1010 may invoke the program code stored in the memory 1030 to execute various video encoding or decoding methods described in the present disclosure. To avoid repetition, details are not described herein again.

[0238] In some embodiments of the present disclosure, the processor 1010 may be a CPU, or the processor 1010 may be another general -purpose processor, a DSP, an ASIC, an FPGA or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or the like. The general -purpose processor may be a microprocessor, or the processor may be another processor or the like.

[0239] The memory 1030 may include a ROM device or a RAM device. Any other proper type of storage device may also be used as the memory 1030. The memory 1030 may include code and data 1031 accessed by the processor 1010 by using a bus 1050. The memory 1030 may further include an operating system 1033 and an application program 1035, and the application program 1035 includes at least one program that allows the processor 1010 to execute the video encoding or decoding method described in the present disclosure. For example, the application program 1035 may include applications 1 to N, which further include a video encoding or decoding application (referred to as a video coding application) performing the video encoding or decoding method described in the present disclosure.

[0240] In addition to a data bus, the bus system 1050 may further include a power bus, a control bus, a status signal bus, and the like. However, for clear description, various types of buses in the figure are marked as the bus system 1050.

[0241] For example, the coding device 1000 may further include one or more output devices, for example, a display 1070. In an example, the display 1070 may be a touch display that combines a display and a touch unit that operably senses a touch input. The display 1070 may be connected to the processor 1010 through the bus 1050.

[0242] Although the processor 1010 and the memory 1030 of the device 1000 are depicted in FIG. 10 as integrated into a single unit, another configuration may be used. Running of the processor 1010 may be distributed in a plurality of machines that can be directly coupled (each machine has one or more processors), or in a local area or another network. The memory 1030 may be distributed in a plurality of machines, such as network-based memories or memories in a plurality of machines running the device 1000. Although only a single bus is depicted herein, the bus 1050 of the device 1000 may be formed by a plurality of buses. Further, the memory 1030 may be directly coupled to another component of the device 1000 or may be accessed through a network, and may include a single integrated unit such as a storage card, or a plurality of units such as a plurality of storage cards. Therefore, the device 1000 may be implemented in a plurality of configurations.

[0243] The above describes systems and devices for implementing video coding. A possible scheme of video coding realized by the described systems and devices is illustrated in FIG. 11. For example, at block 1101, an input video is obtained or received, where the input video includes a set of image frames. At block 1102, each image frame in the input video is partitioned into blocks of some sizes, which may be fixed (e.g., 4x4) or variable (e.g., 4x4, 8x4, 4x8, 8x8). At block 1103, a difference between the current block and the reference block may be computed or a difference among sample values in the current block may be computed, that is, inter prediction or intra prediction may be performed on the current block. The current block is the to-be-processed block. At block 1104, transform(s) (e.g., DCT, DWT, Hadamard) may be applied to the difference. At block 1105, transformed coefficients are quantized using a defined QP value (this is the lossy part of the scheme). At block 1106, scanning defines an order in which the quantized coefficients are reordered. At block 1107, entropy compression (e.g., Huffman, CABAC, or some other entropy compression methods) is applied to the reordered coefficients. Operations at blocks 1101-1107 may be performed at the encoder side (such as the encoder disclosed in FIG. 1 A, FIG. IB, FIG. 2, FIG. 4, FIG. 9, or FIG. 10). After block 1107, a compressed bitstream is stored or transmitted.

[0244] At block 1108, entropy decompression which is an inverse of the entropy compression at block 1107 may be applied to the bitstream. At block 1109, the decompressed data may be reordered using the same order defined at block 1105 to restore coefficients, that is, inverse scanning is performed. The coefficients may be dequantized at block 1110. At block 1111, the dequantized coefficients may undergo an inverse transform of the transform of block 1104 to restore the difference. At block 1112, compensation may be performed on the data input from block 1111 to obtain the decoded block. The decoded blocks may be concatenated at block 1113 to obtain the video. At block 1114, the video is output or displayed. Operations at blocks 1108-1114 may be performed at the decoder side (such as the decoder disclosed in FIG. 1A, FIG. IB, FIG. 3, FIG. 7A, FIG. 7B, FIG. 9, or FIG. 10).

[0245] In the above example, the scanning may determine an order in which transform coefficients (or quantized transform coefficients) are sorted to convert a 2D data array of transform coefficients (or quantized transform coefficients) into a ID data array. This allows entropy compression to be performed conveniently and effectively on the ID transform coefficients (or quantized transform coefficients). The efficiency of entropy compression depends on the scanned transform coefficients (or quantized transform coefficients), that is, the scan order of the transform coefficients (or quantized transform coefficients).

[0246] For another example, the scanning may be used in data packing. The scanning may determine an order in which image data are packed. Data in the image block (coding block, prediction block, transform block, quantized transform block, or the like) may be packed in the defined scan order for transmission. The efficiency of compression for transmission depends on the scan order accordingly. For still another example, the scanning may be used in data storage. The scanning may determine an order in which image data are stored. Data in the image block (coding block, prediction block, transform block, quantized transform block, or the like) may be stored in the defined scan order. During storage, redundancy or similar information may be compressed. The efficiency of storage depends on the scan order accordingly. For yet still another example, the scanning may be used in data displaying. The scanning may determine an order in which image data are displayed. Data in the image block (coding block, prediction block, transform block, quantized transform block, or the like) may be displayed in the defined scan order. The efficiency of display depends on the scan order accordingly.

[0247] That is, the scanning affects the efficiency of subsequent processing. Taking compression as an example, for the encoder, one problem is a loss of accuracy and an introduction of artifacts with increasing compression ratios. One metric of accuracy is peaksignal-to-noise ratio (PSNR). Any encoder has a dependence of PSNR on the number of bits representing the compressed video (bitrate). Another metric of accuracy is Bjontegaard delta rate (BD-rate), which represents a change in an area under a PSNR-bitrate curve when comparing two different encoders. The lower the BD-rate, the better the encoder. For the compression technology, even a small decrease in BD-rate is considered progress. Using an appropriate scan order may improve the efficiency of compression and further decrease compression ratios, improving the accuracy and reducing the artifacts.

[0248] Furthermore, the memory consumption for storing scan orders becomes unreasonable. For example, for a block with a size of 4x4, there are 16! possible scan orders (that is, 16 factorial), which requires approximately 44 bits of memory overhead per block (that is, log2(16!) ~ 44 bits). Therefore, appropriate scan orders may be defined for image data processing to reduce memory consumption.

[0249] As mentioned above, in video coding, coding processing is performed on a per-block basis. That is, blocks such as coding blocks, prediction blocks, or transform blocks may be basic processing units. In the present disclosure, in a case where image data includes multiple basic processing units, one type of scanning is performed on each basic processing unit, and another type of scanning is performed among the multiple basic processing units. For ease of description, scanning on the basic processing unit may be referred to as the scan pattern, and scanning among the multiple basic processing units may be referred to as the scan order. It is noted that the scan order and the scan pattern may be interchangeable. A person skilled in the art may use other terms to differentiate between scanning on the basic processing unit and scanning among the multiple basic processing units.

[0250] There are some technologies to define scanning used for scan transform coefficients or quantized transform coefficients.

[0251] In H.264, in an example, transform coefficients or quantized transform coefficients of a block may be scanned in a zigzag pattern. FIG. 12A illustrates a zigzag pattern of a 4x4 block, where a square in FIG. 12A represents a value in the 4x4 block, for example, the zigzag pattern may be starting from first row and first column, to first row and second column, to second row and first column, to third row and first column, to second row and second column, to first row and third column, to first row and fourth column, to second row and third column, to third row and second column, to fourth row and first column, to fourth row and second column, to third row and third column, to second row and fourth column, to third row and fourth column, to fourth row and third column, and to fourth row and fourth column.

[0252] In H.264, in another example, transform coefficients or quantized transform coefficients of a block may be scanned in a field pattern. The field pattern may be used for interlaced coding. FIG. 12B illustrates a field pattern of a 4x4 block, where a square in FIG.12B represents a value in the 4x4 block, for example, the field pattern may be starting from first row and first column, to second row and first column, to first row and second column, to third row and first column, to fourth row and first column, to second row and second column, to third row and second column, to fourth row and second column, to first row and third column, to second row and third column, to third row and third column, to fourth row and third column, to first row and fourth column, to second row and fourth column, to third row and fourth column, and to fourth row and fourth column.

[0253] In H.264, in still another example, 2x2 DC coefficients of the chroma may be scanned in a raster-scan pattern. FIG. 12C illustrates a raster-scan pattern of 2x2 DC coefficients (that is a 2x2 block), where a square in FIG. 12C represents a value in 2x2 DC coefficients, for example, the raster-scan pattern may be starting from first row and first column, to first row and second column, to second row and first column, and to second row and second column.

[0254] In H.264, in yet still another example, for 16x16 or 8x8 blocks, the zigzag pattern is used within 4x4 subblocks and 4x4 subblocks partitioned from 16x16 or 8x8 blocks are scanned in the raster-scan pattern.

[0255] Different predictions may represent different distributions of coefficients. The above patterns in H.264 may not differentiate among different predictions and may not accurately represent different distributions of coefficients. This approach may not effectively improve the efficiency of entropy compression.

[0256] In H.265 / H.266, coefficient scanning is performed in 4^4 subblocks for all block sizes. Three coefficient scanning manners, diagonal up-right, horizontal, and vertical scans are selected for coding transform coefficients. The vertical scan is used when a prediction direction is close to horizontal, the horizontal scan is used when a prediction direction is close to vertical, and the diagonal up-right scan is used for other prediction directions and inter prediction. FIG.13A illustrates a vertical scan pattern of a 4x4 subblock, FIG. 13B illustrates a horizontal scan pattern of a 4x4 subblock, and FIG. 13C illustrates a diagonal up-right scan pattern of a 4x4 subblock, where arrows point to scan directions and refer to the above FIG. 12A-12C.

[0257] Although the above diagonal up-right, horizontal, and vertical scan patterns differentiate among different predictions, these scan patterns may not accurately represent distributions of different predictions and need to be refined to improve the efficiency of subsequent processing.

[0258] Therefore, according to one or more embodiments of the present disclosure, the image data may correspond to one or more scan patterns to represent distribution characteristics of this image data. This approach may more accurately match the previous processing of the image data, such as prediction, and more effectively improve the subsequent processing of the image data, such as compression or entropy compression.

[0259] Reference is now made to FIG. 14, which illustrates a flowchart of an example scanning method for processing image data according to one or more embodiments of the present disclosure. The method 1400 may be performed by an encoder. The method 1400 may also be performed by a module, a circuit, or a chip in the encoder, or by a logical module or software that may implement all or some functions of the encoder. The encoder may be any encoder described in FIG. 1A, FIG. IB, FIG. 2, FIG. 4, FIG. 9, or FIG. 10. For example, the method 1400 is performed by the encoder and includes the steps 1401 and 1402.

[0260] In step 1401, the encoder determines a scan mode based on a prediction mode of the image data and AT image blocks in the image data, where the scan mode indicates one or more scan patterns for the AT image blocks and a scan order among the AT image blocks, and AT is an integer greater than 0.

[0261] The image data may be any luminance component or any chrominance component. In an example, the image data may be an image frame. The image frame may be partitioned into image unit(s) and each partitioned image unit may be further partitioned into image blocks. As a result, M image blocks may be obtained from the image frame, where M is greater than 1. The image unit may be the coding unit, prediction unit, transform unit, quantified transform unit, or the like. Accordingly, the image block may be the coding block, prediction block, transform block, quantified transform block, or the like. The image block may be a suitable processing basic unit for the image data, and processing of the image data may be performed per image block. In another example, the image data may be an image unit partitioned from the image frame. The image unit may be further partitioned into M image blocks, where M is greater than 1. In still another example, the image data may be one image block, that is, Af is 1.

[0262] The scan mode may be used to scan data across the entire image data. Therefore, the scan mode may relate to the prediction mode of the image data and the AT image blocks in the image data. That is, the scan mode may indicate the one or more scan patterns for the AT image blocks according to the prediction mode of the image data, where scanning is performed on each image block based on the indicated one or more scan patterns. The scan mode may further indicate the scan order among the AT image blocks, where scanning is performed among the AT image blocks.

[0263] According to one or more embodiments, the prediction mode includes one or more of: a horizontal prediction for intra prediction; a vertical prediction for intra prediction; a DC prediction for intra prediction; a planar prediction for intra prediction; an angular prediction for intra prediction; or an inter prediction.

[0264] The horizontal prediction is an intra-prediction technique where a value of each pixel in a prediction block is derived from pixel value(s) directly to its left. For example, for a target pixel at position (x, y) within the block, a prediction value is copied from a pixel at position (x-1, y). This mode assumes that there is a strong correlation between horizontally adjacent pixels, making it particularly effective for regions of the image data that exhibit horizontal edges, textures, or uniform variations in the horizontal direction. The vertical prediction is an intra-prediction technique where a value of each pixel in a prediction block is derived from pixel value(s) directly above it. For example, for a target pixel at position (x, y) within the block, a prediction value is copied from a pixel at position (x, y— 1). This mode assumes that there is a strong correlation between vertically adjacent pixels, making it particularly effective for regions of the image data that exhibit vertical edges, textures, or uniform variations in the vertical direction.

[0265] The DC prediction may calculate a prediction value for each pixel in a prediction block by averaging available neighboring pixels, typically from the left and top boundaries of the prediction block. This average value is used to predict all pixels within the prediction block, assuming that the prediction block has relatively uniform content.

[0266] The planar prediction may calculate a prediction value for each pixel in a prediction block by linearly interpolating between reference pixels located at the prediction block’s boundaries. Specifically, it uses both horizontal and vertical linear interpolation to estimate pixel values within the prediction block. The prediction value at each pixel position is derived from a weighted average of the linearly interpolated values from the top and left boundaries.

[0267] The angular prediction is an intra-prediction technique that models the directional texture of a prediction block by extrapolating pixel values from the prediction block’s boundaries along predefined angles. These angles are designed to capture various directional patterns in the image data, such as edges and textures, thereby improving prediction accuracy and compression efficiency.

[0268] The inter prediction may involve predicting pixel values of a prediction block in the current frame by finding similar blocks in one or more reference frames. This is achieved by estimating the motion between frames and compensating for this motion to generate theprediction block. The difference between the original block and the predicted block (the residual) is encoded and transmitted.

[0269] For the one or more scan patterns for the AT image blocks, in a case where the image data includes one image block, one scan pattern may be determined based on the prediction mode.

[0270] In a case where the image data includes multiple image blocks, for example, when the prediction mode includes the horizontal prediction for intra prediction or the vertical prediction for intra prediction, the multiple image blocks may use one scan pattern; and when the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, two or more patterns may be determined for the multiple image blocks.

[0271] For the scan order among the AT image blocks, in a case where the image data includes one image block, the scan order among the AT image blocks may be default or there is no scan order.

[0272] In a case where the image data includes multiple image blocks, the scan order among the AT image blocks is from an upper left image block first and a lower right image block last. This scan order effectively ensures a logical progression in scanning among the A / image blocks and provides a good connection with both the previous and next image data.

[0273] When the image data includes multiple image blocks, the scan order among the M image blocks may depend on Aland the prediction mode of the image data. In an example, the scan order among the M image blocks may be selected from a set of scan orders according to the prediction mode of the image data, for example, when Al equals to 4, the set of scan orders may include: an order from upper left, to upper right, to lower left, and to lower right, an order from upper left, to lower left, to upper right, and to lower right, and the like. When the prediction mode includes the horizontal prediction for intra prediction, the order from upper left, to lower left, to upper right, and to lower right, may be selected, and when the prediction mode includes the vertical prediction for intra prediction, the order from upper left, to upper right, to lower left, and to lower right, may be selected. In another example, the scan order among the AT image blocks may be a predefined order, and the predefined order is upper left, to lower left, to upper right, and to lower right.

[0274] It is noted that, when the image block is the transform block, quantized transform block, or the like, prediction on the image data may be performed before the transform block or quantized transform block is obtained; when the image block is the prediction block, prediction on the image data may be performed to obtain the prediction block; and when the image blockis the coding block, prediction on the image data may be performed as part of the coding process.

[0275] In step 1402, the encoder obtains, based on the scan mode, scanned image data of the image data.

[0276] The image data is in the 2D space and the scanned image data is in the ID space.

[0277] In a case where the image data includes one image block, data in the one image block may be scanned according to the corresponding scan pattern where the scan order is default or there is no scan order.

[0278] In a case where the image data includes multiple image blocks, each image block is selected in turn according to the scan order. The scan pattern corresponding to each selected image block is then used to scan the data within that block. This process continues sequentially until the last image block is scanned.

[0279] For example, when the image block is the transform block, quantized transform block, or the like, the scanned image data includes the scanned transform coefficients or the scanned quantized transform coefficients. For another example, when the image block is the prediction block or the coding process, the scanned image data includes the scanned pixels.

[0280] According to the scanning method for processing image data according to one or more embodiments of the present disclosure, the encoder determines the scan mode based on the prediction mode of the image data and the M image blocks in the image data, where the scan mode indicates the one or more scan patterns for the AT image blocks and the scan order among the A / image blocks, and Af is an integer greater than 0; and encoder obtains, based on the scan mode, scanned image data of the image data. That is, the image data may correspond to one or more scan patterns to represent distribution characteristics of this image data. This method may more accurately match the previous processing of the image data, such as prediction, and more effectively improve the subsequent processing of the image data, such as compression or entropy compression.

[0281] According to one or more embodiments, the scan mode further indicates a scan pattern in the one or more scan patterns to be used for each image block in the AT image blocks. That is, for each image block in the A / image blocks, a corresponding scan pattern may be indicated. Therefore, data in each image block may be scanned in the more suitable scan pattern.

[0282] In an example, for the horizontal prediction for intra prediction or the vertical prediction for intra prediction, a unified scan pattern may be indicated for all image blocks in the image data. This unified scan pattern may be predefined.

[0283] In another example, for the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, a scan pattern may be indicated for each image block in the image data. For example, when AT equals to 4, a first scan pattern is indicated for the image block at first row and first column, a second scan pattern is indicated for the image block at second row and first column, a third scan pattern is indicated for the image block at first row and second column, and a fourth scan pattern is indicated for the image block at second row and second column. It is noted that the first, second, third, and fourth scan patterns are used to mark scan patterns of different image blocks and may not indicate sequence information among different image blocks. In an example, the first scan pattern and the second scan pattern may be the same, and the third scan pattern and the fourth scan pattern may be the same. In another example, the first scan pattern and the third scan pattern may be the same, and the second scan pattern and the fourth scan pattern may be the same. In still another example, the first, second, third, and fourth scan patterns are different. In yet still another example, the second scan pattern and the third scan pattern may be the same and the remaining scan patterns may be different, or the first scan pattern and the fourth scan pattern may be the same and the remaining scan patterns may be different.

[0284] According to one or more embodiments, step 1402 may include the following. The encoder determines the scan order among the M image blocks; and selects the one or more scan patterns from a set of scan patterns according to the prediction mode of the image data. The process of determining the scan order among the M image blocks may refer to the related description as above. The set of scan patterns may be described below.

[0285] According to one or more embodiments, the set of scan patterns may include one or more of: an extended horizontal scan pattern; an extended vertical scan pattern; an extended zigzag scan pattern; or an extended diagonal scan pattern.

[0286] The Extended Horizontal Scan Pattern

[0287] In one or more embodiments, the horizontal scan pattern may be extended or enhanced to obtain the extended horizontal scan pattern. The extended horizontal scan pattern is a combination of a horizontal scan pattern and a diagonal scan pattern. For example, each image block in the M image blocks is a N*N block, N is an integer equal to or greater than 4, and the extended horizontal scan pattern includes: the horizontal scan pattern of scanning part or all elements in N rows in the N*N block; and the diagonal scan pattern of scanning remaining elements in the A*Ablock. In an example, N / 4 or A72 or 2N / 3 rows are scanned in the horizontal direction and the remaining rows are scanned in an interspersed manner using both horizontal and diagonal directions. In another example, the first row or the last row is scanned in thehorizontal direction, part elements in the remaining rows are scanned in the horizontal direction, and the remaining elements are scanned in the diagonal direction.

[0288] For example, when N equals to 4, and the extended horizontal scan pattern, as disclosed in FIG. 15 A, includes: scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in the first row and a third column, to an element in the first row and a fourth column; to an element in a second row and the first column, to an element in the second row and the second column, to an element in the second row and the third column; to an element in a third row and the first column, to an element in the second row and the fourth column; to an element in the third row and the second column, to an element in the third row and the third column; to an element in a fourth row and the first column, to an element in the third row and the fourth column; to an element in the fourth row and the second column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

[0289] In the above example, the 4*4 block may be used as the basic processing unit. For the hardware architecture of the encoder, there is a problem how friendly the encoder’s algorithm is for the hardware architecture of the encoder. For some chips, a lower amount of data processing is preferable. For example, scanning larger blocks of a size 8x8 as a whole may result in a loss of performance compared to blocks of a size 4x4. In this situation, an algorithm oriented toward 4x4 blocks may potentially have an advantage on a specific chip compared to one that needs simultaneous access to data in 8x8 blocks. For example, 4x4 blocks may provide improving of BD-rate on 0.2-0.6% compared to H.264 scanning schemes.

[0290] Taking the transform block as an example, when the transform coefficients in the transform block are scanned by this extended horizontal scan pattern, the entropy compression efficiency of video coding may be improved. Using this extended horizontal scan pattern, a trade-off between compression performance and hardware efficiency may be optimal.

[0291] The Extended Vertical Scan Pattern

[0292] In one or more embodiments, the vertical scan pattern may be extended or enhanced to obtain the extended vertical scan pattern. The extended vertical scan pattern is a combination of a vertical scan pattern and a diagonal scan pattern. For example, each image block in the AT image blocks is a V*V block, Vis an integer equal to or greater than 4, and the extended vertical scan pattern includes: the vertical scan pattern of scanning part or all elements in N columns in the N*N block; and the diagonal scan pattern of scanning remaining elements in the N*N block. In an example, N / 4 or N / 2 or 2N / 3 columns are scanned in the vertical direction and the remaining columns are scanned in an interspersed manner using both vertical and diagonaldirections . In another example, the first column or the last column is scanned in the vertical direction, part elements in the remaining columns are scanned in the vertical direction, and the remaining elements are scanned in the diagonal direction.

[0293] For example, when N equals to 4, and the extended vertical scan pattern, as disclosed in FIG. 15B, includes: scanning starting from an element in a first row and a first column, to an element in a second row and the first column, to an element in a third row and the first column, to an element in a fourth row and the first column; to an element in the first row and a second column, to an element in the second row and the second column, to an element in the third row and the second column; to an element in the first row and a third column, to an element in the fourth row and the second column; to an element in the second row and the third column, to an element in the third row and the third column; to an element in the first row and the fourth column, to an element in the fourth row and the third column; to an element in the second row and the fourth column, to an element in the third row and the fourth column; and to element in the fourth row and the fourth column.

[0294] Similar to the extended horizontal scan pattern, using this extended vertical scan pattern, a trade-off between compression performance and hardware efficiency may also be optimal.

[0295] The Extended Zigzag Scan Pattern

[0296] In one or more embodiments, the zigzag scan pattern may be extended or enhanced to obtain the extended zigzag scan pattern. The extended zigzag scan pattern is a combination of a raster scan pattern and a zigzag scan pattern. For example, each image block in the AT image blocks is a N*N block, N is an integer equal to or greater than 4, and the extended zigzag scan pattern includes: the raster scan pattern of scanning starting from an element in a first row and a first column in the N*N block to an element in a second row and a second column in the N*N block; and the zigzag scan pattern of scanning remaining elements in the N*N block, where the zigzag scan pattern follows a scanning trend of the raster scan pattern. That is, regions in the image data, which are located at the upper left in the image data, may be scanned by the raster scan pattern, and the remaining regions in the image data may be scanned by the zigzag scan pattern. This scan pattern may be more suitable to other predictions in addition to the horizontal prediction for intra prediction and the vertical prediction for intra prediction.

[0297] For example, when N equals to 4, and the extended zigzag scan pattern, as disclosed in FIG. 15C, includes: scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in a second row and the first column, to an element in the second row and the second column; to an element in a third row and the first column, to an element in the first row and a third column; to an element in the first rowand a fourth column, to an element in the second row and the third column, to an element in the third row and the second column, to an element in a fourth row and the first column, to an element in the fourth row and the second column; to an element in the third row and the third column, to an element in the second row and the fourth column, to an element in the third row and the fourth column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

[0298] Similar to the extended horizontal or vertical scan pattern, using this extended zigzag scan pattern, a trade-off between compression performance and hardware efficiency may also be optimal.

[0299] The Extended Diagonal Scan Pattern

[0300] In one or more embodiments, the diagonal scan pattern may be extended or enhanced to obtain the extended diagonal scan pattern. The extended diagonal scan pattern is a combination of a raster scan pattern and a diagonal scan pattern. For example, each image block in the A / image blocks is aTVWblock, TVis an integer equal to or greater than 4, and the extended diagonal scan pattern includes: the raster scan pattern of scanning starting from an element in a first row and a first column in the N*N block to an element in a second row and a second column in the N*N block; and the diagonal scan pattern of scanning remaining elements in the N*N block in the diagonal scan pattern, where the diagonal scan pattern follows a scanning trend of the raster scan pattern. That is, regions in the image data, which are located at the upper left or at the lower right in the image data, may be scanned by the raster scan pattern, and the remaining regions in the image data may be scanned by the diagonal scan pattern. This scan pattern may be more suitable to the prediction for intra prediction in addition to the horizontal prediction for intra prediction and the vertical prediction for intra prediction.

[0301] For example, when N equals to 4, and the extended diagonal scan pattern, as disclosed in FIG. 15D, includes: scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in a second row and the first column, to an element in the second row and the second column; to an element in the first row and a third column, to an element in a third row and the first column; to an element in the first row and a fourth column, to an element in the second row and the third column, to an element in the third row and the second column, to an element in a fourth row and the first column; to an element in the second row and the fourth column, to an element in the third row and the third column, to an element in the fourth row and the second column; and to an element in the third row and the fourth column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

[0302] Similar to the extended horizontal or vertical pattern or zigzag scan pattern, using this extended diagonal scan pattern, a trade-off between compression performance and hardware efficiency may also be optimal.

[0303] The following describes some examples.

[0304] In an example, there is one 4*4 block, when the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern as shown in FIG. 15A is used. When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern as shown in FIG. 15B is used. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended zigzag scan pattern as shown in FIG. 15C is used, or the extended diagonal scan pattern as shown in FIG. 15D is used.

[0305] In another example, there are 24*4 blocks.

[0306] In case 1, the 2 4*4 blocks are arranged horizontally. When the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern as shown in FIG. 15A is used and the scanning on the 24*4 blocks is shown in FIG. 16A, where block 0 is scanned first and block 1 is scanned last. When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern as shown in FIG.15B is used and the scanning on the 2 4*4 blocks is shown in FIG. 16B, where block 0 is scanned first and block 1 is scanned last. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended zigzag scan pattern as shown in FIG. 15C is used and the scanning on the 24*4 blocks is shown in FIG. 16C, where block 0 is scanned first and block 1 is scanned last. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended diagonal scan pattern as shown in FIG. 15D is used and the scanning on the 2 4*4 blocks is shown in FIG. 16D, where block 0 is scanned first and block 1 is scanned last.

[0307] For the example in FIG. 16C, block 1 may be also scanned by the extended horizontal scan pattern such as shown in FIG. 16E, or the extended vertical scan pattern such as shown in FIG. 16F, or the extended diagonal scan pattern such as shown in FIG. 16G.

[0308] For the example in FIG. 16D, block 1 may be also scanned by the extended horizontal scan pattern such as shown in FIG. 16H, or the extended vertical scan pattern such as shown in FIG. 161, or the extended zigzag scan pattern such as shown in FIG. 16 J.

[0309] In case 2, the 24*4 blocks are arranged vertically. When the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern as shown in FIG. 15A is used and the scanning on the 24*4 blocks is shown in FIG. 17A, where block 0 is scanned first and block 1 is scanned last. When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern as shown in FIG. 15B is used and the scanning on the 24*4 blocks is shown in FIG. 17B, where block 0 is scanned first and block 1 is scanned last. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended zigzag scan pattern as shown in FIG. 15C is used and the scanning on the 24*4 blocks is shown in FIG. 17C, where block 0 is scanned first and block 1 is scanned last. When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended diagonal scan pattern as shown in FIG. 15D is used and the scanning on the 24*4 blocks is shown in FIG. 17D, where block 0 is scanned first and block 1 is scanned last.

[0310] For the example in FIG. 17C, block 1 may be also scanned by the extended horizontal scan pattern such as shown in FIG. 17E, or the extended vertical scan pattern such as shown in FIG. 17F, or the extended diagonal scan pattern such as shown in FIG. 17G.

[0311] For the example in FIG. 17D, block 1 may be also scanned by the extended horizontal scan pattern such as shown in FIG. 17H, or the extended vertical scan pattern such as shown in FIG. 171, or the extended zigzag scan pattern such as shown in FIG. 17 J.

[0312] In still another example, there are 44*4 blocks.

[0313] In some embodiments, the scan mode indicates one extended horizontal scan pattern or one extended vertical horizontal scan pattern, the scan order is from the upper left image block, to the lower left image block, to the upper right image block, and to the lower right image block.

[0314] In other embodiments, the scan mode indicates one extended horizontal scan pattern or one extended vertical horizontal scan pattern, the scan order is from an upper left image block, to an upper right image block, to a lower left image block, and to a lower right image block.

[0315] When the prediction mode includes the vertical prediction for intra prediction, the extended horizontal scan pattern as shown in FIG. 15A is used and the scanning on the 44*4 blocks is shown in FIG. 18 A, where block 0 is scanned first, block 1 is scanned following block 0, block 2 is scanned following block 1, and block 3 is scanned last. Alternatively, the extended horizontal scan pattern as shown in FIG. 15A is used and the scanning on the 44*4 blocks isshown in FIG. 18B, where block 0 is scanned first, block 2 is scanned following block 0, block 1 is scanned following block 2, and block 3 is scanned last.

[0316] When the prediction mode includes the horizontal prediction for intra prediction, the extended vertical scan pattern as shown in FIG. 15B is used and the scanning on the 4 4*4 blocks is shown in FIG. 18C, where block 0 is scanned first, block 1 is scanned following block 0, block 2 is scanned following block 1, and block 3 is scanned last. Alternatively, the extended vertical scan pattern as shown in FIG. 15B is used and the scanning on the 4 4*4 blocks is shown in FIG. 18D, where block 0 is scanned first, block 2 is scanned following block 0, block 1 is scanned following block 2, and block 3 is scanned last.

[0317] In some embodiments the scan mode indicates hybrid scan patterns, and the scan order is from an upper left image block using an extended zigzag scan pattern, to a lower left image block using an extended horizontal scan pattern, to an upper right image block using an extended vertical scan pattern, to a lower right image block using the extended zigzag scan pattern.

[0318] In other embodiments, the scan mode indicates hybrid scan patterns, and the scan order is from an upper left image block using an extended zigzag scan pattern, to an upper right image block using an extended vertical scan pattern, to a lower left image block using an extended horizontal scan pattern, and to a lower right image block using the extended zigzag scan pattern.

[0319] When the prediction mode includes the DC prediction for intra prediction, the planar prediction for intra prediction, the angular prediction for intra prediction, or the inter prediction, the extended horizontal scan pattern as shown in FIG. 15 A, the extended vertical scan pattern as shown in FIG. 15B, and the extended zigzag scan pattern as shown in FIG. 15C are used, and the scanning on the 4 4*4 blocks is shown in FIG. 18E, where block 0 is scanned first, block 1 is scanned following block 0, block 2 is scanned following block 1, and block 3 is scanned last. Alternatively, as shown in FIG. 18F, block 0 is scanned first, block 2 is scanned following block 0, block 1 is scanned following block 2, and block 3 is scanned last.

[0320] Alternatively, the extended zigzag scan pattern may be replaced by the extended diagonal scan pattern as shown in FIG. 18G and shown in FIG. 18H.

[0321] Alternatively, in FIG. 18E, FIG. 18F, FIG. 18G, and FIG. 18H, the extended vertical scan pattern at the upper right and the extended horizontal scan pattern at the lower left may be changed.

[0322] Alternatively, in FIG. 18E, FIG. 18F, FIG. 18G, and FIG. 18H, the extended vertical scan pattern may be used both at the upper right and at the lower left.

[0323] Alternatively, in FIG. 18E, FIG. 18F, FIG. 18G, and FIG. 18H, the extended horizontal scan pattern may be used both at the upper right and at the lower left.

[0324] The above describes actions in the encoder side and the following describes actions in the decoder.

[0325] Reference is now made to FIG. 19, which illustrates a flowchart of an example scanning method for processing image data according to one or more embodiments of the present disclosure. The method 1900 may be performed by a decoder. The method 1900 may also be performed by a module, a circuit, or a chip in the decoder, or by a logical module or software that may implement all or some functions of the decoder. The decoder may be any decoder shown in FIG. 1A, FIG. IB, FIG. 3, FIG. 7A, FIG. 7B, FIG. 9, or FIG. 10. For example, the method 1900 is performed by the encoder and includes the steps 1901 and 1902.

[0326] In step 1901, the decoder obtains scanned image data and a scan mode, where the scan mode is determined based on a prediction mode of the image data and AT image blocks in the image data, where the scan mode indicates one or more scan patterns for the M image blocks and a scan order among the AT image blocks, and AT is an integer greater than 0.

[0327] The decoder may receive a bitstream and parse the bitstream to obtain the scanned image data and the scan mode.

[0328] Alternatively, the decoder may read the scanned image data and the scan mode from the memory.

[0329] In step 1902, the decoder recovers, based on an inverse of the scan mode, the image data from the scanned image data. That is, the inverse of the scan mode decoder may refer to an inverse scan mode.

[0330] The decoder may replace the data in the scanned image data according to the inverse scan mode to recover the image data. That is, the ID data array may be recovered to the 2D data array.

[0331] Reference may be made to the same part at the encoder side, which is not repeated herein.

[0332] In various embodiments of the present disclosure, without special instructions and logical conflicts, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0333] It will be understood that, in order to achieve the above functions, the encoder and the decoder each include corresponding hardware and / or software modules for implementing various functions. A person skilled in the art should easily realize that the embodiments of thepresent disclosure can be implemented in the form of a hardware or a combination of hardware and computer software in combination with the units and algorithm steps described in the embodiments of the present disclosure. Whether a certain function is executed by hardware or by computer software driving hardware depends on the specific application and design constraint conditions of the technical solution.

[0334] The above describes the scanning method for processing image data according to one or more embodiments of the present disclosure. The following describes the scanning apparatus for processing image data according to one or more embodiments of the present disclosure.

[0335] Reference is now made to FIG. 20, in a first implementation manner, the apparatus 2000 may be applied to the encoder described above and realize the above method implementations performed by the encoder. The apparatus 2000 may be implemented as the encoder or a module (for example, a chip or a circuit) in the encoder or at the side of the encoder.

[0336] In some embodiments, the apparatus 2000 includes the processing unit 2012. The processing unit 2012 is configured to determine a scan mode based on a prediction mode of the image data and AT image blocks in the image data, where the scan mode indicates one or more scan patterns for the AT image blocks and a scan order among the AT image blocks, and AT is an integer greater than 0; and obtain, based on the scan mode, scanned image data of the image data.

[0337] In some embodiments, the processing unit 2012 is configured to determine the scan order among the M image blocks; and select the one or more scan patterns from a set of scan patterns according to the prediction mode of the image data.

[0338] In some embodiments, the apparatus 2000 further includes a storage unit 2011 configured to store the image data and / or the scanned image data.

[0339] In some embodiments, the apparatus 2000 further includes a communication unit 2013 configured to receive the image data and / or transmit the scanned image data.

[0340] Reference is now made to FIG. 20 again, in a second implementation manner, the apparatus 2000 may be applied to the decoder described above and realize the above method implementations performed by the decoder. The apparatus 2000 may be implemented as the decoder or a module (for example, a chip or a circuit) in the decoder or at the side of the decoder.

[0341] In some embodiments, the apparatus 2000 includes the processing unit 2012. The processing unit 2012 is configured to obtain scanned image data and a scan mode, where the scan mode is determined based on a prediction mode of the image data and AT image blocks in the image data, where the scan mode indicates one or more scan patterns for the A / image blocksand a scan order among the AT image blocks, and AT is an integer greater than 0; and recover, based on an inverse of the scan mode, the image data from the scanned image data.

[0342] In some embodiments, the storage unit 2011 is configured to store the image data and / or the scanned image data.

[0343] In some embodiments, the communication unit 2013 configured to transmit the image data and / or receive the scanned image data.

[0344] For the first or second implementation manner, in some embodiments, the scan mode further indicates a scan pattern in the one or more scan patterns to be used for each image block in the AT image blocks.

[0345] For the first or second implementation manner, the prediction mode may refer to the description in the method embodiments at the encoder side.

[0346] For the first or second implementation manner, the set of scan patterns, extended horizontal scan pattern, extended vertical scan pattern, extended zigzag scan pattern, and extended diagonal scan pattern may refer to the description in the method embodiments at the encoder side.

[0347] For the first or second implementation manner, the scan order may refer to the description in the method embodiments at the encoder side.

[0348] Referring back to FIG. 9, in a third implementation manner, the video coding device 900 may be applied to the encoder and realize the above method embodiments performed by the encoder. The processor 930 executes computer executable instructions stored in the memory 960 for executing the above methods implemented by the encoder.

[0349] Referring back to FIG. 9 again, in a fourth implementation manner, the video coding device 900 may be applied to the decoder and realize the above method embodiments performed by the decoder. The processor 930 executes computer executable instructions stored in the memory 960 for executing the above methods implemented by the decoder.

[0350] In some embodiments, the system including the encoder for realizing the method performed by the encoder and the decoder for realizing the method performed by the decoder may be provided.

[0351] For instance, it is understood that a disclosure in connection with a described method may also hold true for a corresponding apparatus, device, or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding apparatus, device, or system may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more ofthe plurality of steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific apparatus, device, or system is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the drawings. Further, it is understood that the features of the various implementations and / or aspects described herein may be combined with each other, unless specifically noted otherwise.

[0352] One or more embodiments of the present disclosure provide a computer-readable storage medium (e.g. a non-transitory computer-readable storage medium). The computer-readable storage medium has stored thereon program instructions that, when run on an encoder and / or decoder, cause the encoder and / or decoder to execute one or more steps of the scanning methods for processing image data as described in any one of the above embodiments.

[0353] For example, the computer-readable storage medium includes, but is not limited to, a magnetic storage device (e.g. a hard disk, a floppy disk or a magnetic tape), an optical disk (e.g. a compact disk (CD), or a DVD), a smart card, and a flash memory device (e.g. an erasable programmable read-only memory (EPROM), a card, a stick or a key driver). Various computer-readable storage media described in the embodiments of the present disclosure may represent one or more devices and / or other machine-readable storage media, which are used for storing information. The term “computer-readable storage medium” may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0354] One or more embodiments of the present disclosure further provide a computer program product. The computer program product includes program instructions carried on a non-transitory computer-readable storage medium. When executed on a encoder and / or decoder, the computer program instructions cause the encoder and / or decoder to perform one or more steps of the scanning methods for processing image data as described in any one of the above embodiments.

[0355] Beneficial effects of the computer-readable storage medium and the computer program product are the same as the beneficial effects of the method for communication as described in some of the above embodiments, and details will not be repeated here.

[0356] One or more embodiments of the present disclosure provide a computer program including instructions. The instructions, when executed by a processor, may cause the processor to implement a method of the present disclosure.

[0357] One or more embodiments of the present disclosure provide an integrated circuit. The integrated circuit includes one or more logic circuits for executing the steps of the method for communication of the present disclosure.

[0358] One or more embodiments of the present disclosure provide an apparatus including means (e.g. at least one processor) to implement a method of the present disclosure. The apparatus may be device (that is, a terminal device or a network device) or a module or component in the device. The at least one processor may execute instructions stored in a computer-readable medium to implement the method.

[0359] The apparatus may be a device or an apparatus implemented in a device. For example, the apparatus implemented in a device may be an integrated circuit, which in some contexts may be known by other colloquial names, such as chip, modem, modem chip, baseband chip, or baseband processor. In some embodiments, one or more integrated circuits can be packaged into a system-on-chip, a system-in-package, or a multi-chip module. The apparatus may include one or more integrated circuits or include one or more integrated circuits and other discrete components.

[0360] In the present disclosure, the terms “a” or “an” are defined to mean “at least one”, that is, these terms do not exclude a plural number of items, unless stated otherwise.

[0361] In the present disclosure, terms such as “substantially”, “generally” and “about”, which modify a value, condition or characteristic of a feature of an example embodiment, should be understood to mean that the value, condition or characteristic is defined within tolerances that are acceptable for the proper operation of the example embodiment for its intended application.

[0362] In the present disclosure, unless stated otherwise, the terms “connected” and “coupled”, and derivatives and variants thereof, refer herein to any structural or functional connection or coupling, either direct or indirect, between two or more elements. For example, the connection or coupling between the elements can be acoustical, mechanical, optical, electrical, thermal, logical, or any combinations thereof.

[0363] In the present disclosure, expressions such as “match”, “matching” and “matched”, including variants and derivatives thereof, are intended to refer herein to a condition in which two or more elements are either the same or within some predetermined tolerance of each other. That is, these terms are meant to encompass not only “exactly” or “identically” matching the two elements but also “substantially”, “approximately” or “subjectively” matching the two ormore elements, as well as providing a higher or best match among a plurality of matching possibilities.

[0364] In the present disclosure, the expression “based on” is intended to mean “based at least partly on”, that is, this expression can mean “based solely on” or “based partially on”, and so should not be interpreted in a limited manner. More particularly, the expression “based on” could also be understood as meaning “depending on”, “representative of’, “indicative of’, “associated with” or similar expressions.

[0365] In the present disclosure, unless otherwise specified, ordinal numbers such as “first” and “second” in embodiments of the present disclosure are used to distinguish between a plurality of objects, and are not used to limit a sequence, a time sequence, priorities, or importance of the plurality of objects.

[0366] A person skilled in the art should understand that embodiments of the present disclosure may be provided as a method, an apparatus (or system), computer-readable storage medium, or a computer program product. Therefore, the present disclosure may use a form of a hardware-only embodiment, a software-only embodiment, or one or more embodiments with a combination of software and hardware. Moreover, the present disclosure may use a form of a computer program product that is implemented on one or more computer-usable storage media (including but not limited to a disk memory, an optical memory, and the like) that include computer-usable program code.

[0367] This disclosure is described with reference to the flowcharts and / or block diagrams of the method, the device (system), and the computer program product according to the present disclosure. It should be understood that computer program instructions may be used to implement each process and / or each block in the flowcharts and / or the block diagrams and a combination of a process and / or a block in the flowcharts and / or the block diagrams. The computer program instructions may be provided for a general -purpose computer, a dedicated computer, an embedded processor, or a processor of another programmable data processing device and enable a machine to execute the instructions. When executed by any computer or the processor of a programmable data processing device, the instructions cause the apparatus to implement specific functions as described in one or more procedures in the flowcharts and / or one or more blocks in the block diagrams. The computer program instructions may alternatively be stored in a computer-readable memory that can indicate a computer or another programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate an artifact that includes an instructionapparatus. The instruction apparatus implements a specific function in one or more procedures in the flowcharts and / or one or more blocks in the block diagrams.

[0368] The computer program instructions may alternatively be loaded onto a computer or another programmable data processing device, so that a series of operations and steps are performed on the computer or the another programmable device, so that computer-implemented processing is generated. Therefore, the instructions executed on the computer or on another programmable device provide steps for implementing specific functions as described in one or more procedures in the flowcharts and / or one or more blocks in the block diagrams.

[0369] It is clear that a person skilled in the art can make various modifications and variations to the present disclosure without departing from the scope of this disclosure. This disclosure is intended to cover these modifications and variations of the present disclosure provided that they fall within the scope of protection defined by the following claims and their equivalent technologies.

Claims

CLAIMS1. A scanning method for processing image data, the method comprising: determining a scan mode based on a prediction mode of the image data and image blocks in the image data, wherein the scan mode indicates one or more scan patterns for the M image blocks and a scan order among the M image blocks, and is an integer greater than 0; and obtaining, based on the scan mode, scanned image data of the image data.

2. The method of claim 1, wherein the scan mode further indicates a scan pattern in the one or more scan patterns to be used for each image block in the M image blocks.

3. The method of claim 1 or 2, wherein the prediction mode comprises one or more of: a horizontal prediction for intra prediction;a vertical prediction for intra prediction;a direct current (DC) prediction for intra prediction;a planar prediction for intra prediction;an angular prediction for intra prediction; oran inter prediction.

4. The method of any one of claims 1 to 3, wherein the determining a scan mode comprises: determining the scan order among the M image blocks; andselecting the one or more scan patterns from a set of scan patterns according to the prediction mode of the image data.

5. The method of claim 4, wherein the set of scan patterns comprise one or more of: an extended horizontal scan pattern;an extended vertical scan pattern;an extended zigzag scan pattern; oran extended diagonal scan pattern.

6. The method of claim 5, wherein:the extended horizontal scan pattern is a combination of a horizontal scan pattern and a diagonal scan pattern;the extended vertical scan pattern is a combination of a vertical scan pattern and a diagonal scan pattern;the extended zigzag scan pattern is a combination of a raster scan pattern and a zigzag scan pattern; andthe extended diagonal scan pattern is a combination of a raster scan pattern and a diagonal scan pattern.

7. The method of claim 6, wherein each image block in the A / image blocks is ablock, TV is an integer equal to or greater than 4, and the extended horizontal scan pattern comprises:the horizontal scan pattern of scanning part or all elements in N rows in the N*N block; and the diagonal scan pattern of scanning remaining elements in the N*N block.

8. The method of claim 7, wherein N equals to 4, and the extended horizontal scan pattern comprises:scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in the first row and a third column, to an element in the first row and a fourth column;to an element in a second row and the first column, to an element in the second row and the second column, to an element in the second row and the third column;to an element in a third row and the first column, to an element in the second row and the fourth column;to an element in the third row and the second column, to an element in the third row and the third column;to an element in a fourth row and the first column, to an element in the third row and the fourth column;to an element in the fourth row and the second column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

9. The method of claim 6, wherein each image block in the A / image blocks is ablock, N is an integer equal to or greater than 4, and the extended vertical scan pattern comprises: the vertical scan pattern of scanning part or all elements in N columns in the N*N block; andthe diagonal scan pattern of scanning remaining elements in the N*N block.

10. The method of claim 9, wherein N equals to 4, and the extended vertical scan pattern comprises:scanning starting from an element in a first row and a first column, to an element in a second row and the first column, to an element in a third row and the first column, to an element in a fourth row and the first column;to an element in the first row and a second column, to an element in the second row and the second column, to an element in the third row and the second column;to an element in the first row and a third column, to an element in the fourth row and the second column;to an element in the second row and the third column, to an element in the third row and the third column;to an element in the first row and the fourth column, to an element in the fourth row and the third column;to an element in the second row and the fourth column, to an element in the third row and the fourth column; and to element in the fourth row and the fourth column.

11. The method of claim 6, wherein each image block in the A / image blocks is a N*N block, N is an integer equal to or greater than 4, and the extended zigzag scan pattern comprises: the raster scan pattern of scanning starting from an element in a first row and a first column in the N*N block to an element in a second row and a second column in the N*N block; and the zigzag scan pattern of scanning remaining elements in the N*N block, wherein the zigzag scan pattern follows a scanning trend of the raster scan pattern.

12. The method of claim 11, wherein N equals to 4, and the extended zigzag scan pattern comprises:scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in a second row and the first column, to an element in the second row and the second column;to an element in a third row and the first column, to an element in the first row and a third column;to an element in the first row and a fourth column, to an element in the second row and the third column, to an element in the third row and the second column, to an element in a fourth row and the first column, to an element in the fourth row and the second column;to an element in the third row and the third column, to an element in the second row and the fourth column, to an element in the third row and the fourth column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

13. The method of claim 6, wherein each image block in the A / image blocks is ablock, N is an integer equal to or greater than 4, and the extended diagonal scan pattern comprises:the raster scan pattern of scanning starting from an element in a first row and a first column in the N*N block to an element in a second row and a second column in the N*N block; and the diagonal scan pattern of scanning remaining elements in the N*N block, wherein the diagonal scan pattern follows a scanning trend of the raster scan pattern.

14. The method of claim 13, wherein N equals to 4, and the extended diagonal scan pattern comprises:scanning starting from an element in a first row and a first column, to an element in the first row and a second column, to an element in a second row and the first column, to an element in the second row and the second column;to an element in the first row and a third column, to an element in a third row and the first column;to an element in the first row and a fourth column, to an element in the second row and the third column, to an element in the third row and the second column, to an element in a fourth row and the first column;to an element in the second row and the fourth column, to an element in the third row and the third column, to an element in the fourth row and the second column;to an element in the third row and the fourth column, to an element in the fourth row and the third column, and to an element in the fourth row and the fourth column.

15. The method of any one of claims 1 to 14, wherein the scan order among the AT image blocks is from an upper left image block first and a lower right image block last.

16. The method of claim 15, wherein in a case where AT equals to 4 and the scan mode indicates one extended horizontal scan pattern or one extended vertical horizontal scan pattern, the scan order is from an upper left image block, to a lower left image block, to an upper right image block, and to a lower right image block.

17. The method of claim 15, wherein in a case where M equals to 4 and the scan mode indicates one extended horizontal scan pattern or one extended vertical horizontal scan pattern, the scan order is from an upper left image block, to an upper right image block, to a lower left image block, and to a lower right image block.

18. The method of claim 15, wherein in a case where AT equals to 4 and the scan mode indicates hybrid scan patterns, the scan order is from an upper left image block using an extended zigzag scan pattern, to a lower left image block using an extended horizontal scan pattern, to an upper right image block using an extended vertical scan pattern, to a lower right image block using the extended zigzag scan pattern.

19. The method of claim 15, wherein in a case where AT equals to 4 and the scan mode indicates hybrid scan patterns, the scan order is from an upper left image block using an extended zigzag scan pattern, to an upper right image block using an extended vertical scan pattern, to a lower left image block using an extended horizontal scan pattern, and to a lower right image block using the extended zigzag scan pattern.

20. A scanning method for processing image data, the method comprising:obtaining scanned image data and a scan mode, wherein the scan mode is determined based on a prediction mode of the image data and A / image blocks in the image data, wherein the scan mode indicates one or more scan patterns for the AT image blocks and a scan order among the AT image blocks, and AT is an integer greater than 0; andrecovering, based on an inverse of the scan mode, the image data from the scanned image data.

21. An apparatus for processing image data, configured to perform the method according to any one of claims 1 to 19 or claim 20.

22. A computer-readable storage medium having instructions stored thereon which, when executed by an apparatus, cause the apparatus to perform the method of any one of claims 1 to 19 or claim 20.

23. A computer program product storing instructions which, when executed, cause an apparatus to perform the method of any one of claims 1 to 19 or claim 20.

Images