Image encoding device, image decoding device, image encoding method, image decoding method, and program

Abstract

An image encoding device according to the present invention comprises: a first division means that carries out a division into a plurality of blocks; a second division means that divides each block into a plurality of sub-blocks; a prediction means that calculates a prediction error on the basis of a prediction image generated by executing a prediction process in units of sub-blocks; a quantization means that quantizes the prediction error using a quantization parameter and generates a quantization coefficient; and an encoding means that encodes the quantization coefficient and encodes the quantization parameter. When the prediction means has divided sub-blocks into four sub-blocks, the quantization means determines whether to share the quantization parameter among the four sub-blocks on the basis of a comparison between size information corresponding to the sizes of the four sub-blocks and a quantization control threshold indicating a unit for encoding the quantization parameter, and on the basis of the determination thereof, the encoding means switches the process of encoding the quantization parameter of the four sub-blocks.

Description

Image Encoding Device, Image Decoding Device, Image Encoding Method, Image Decoding Method, and Program

[0001] The present disclosure relates to an image encoding device, an image decoding device, an image encoding method, an image decoding method, and a program.

[0002] As an encoding method for compression recording of moving images, the VVC (Versatile Video Coding) encoding method is known. In VVC, for improving encoding efficiency, a basic block called CTU (Coding Tree Unit) is divided into sub-blocks of rectangles as well as conventional squares. The CTU is up to 128×128 pixels. Patent Document 1 discloses a technique for calculating the size of a sub-block for encoding a quantization parameter and determining whether to encode the quantization parameter according to the size.

[0003] In recent years, in JVET (Joint Video Experts Team) that standardized VVC, new encoding techniques for achieving encoding efficiency and higher image quality beyond VVC are being studied.

[0004] In VVC, squares and rectangles are used as the shapes of sub-blocks generated by block division. Also, the area of the block before division occupying the image is filled without gaps by a combination of square or rectangular sub-blocks by dividing the block into a plurality of sub-blocks.

[0005] Japanese Patent Application Laid-Open No. 2012-161074

[0006] However, when dividing sub-blocks and encoding the quantization parameters of all sub-blocks, the load of the encoding process increases as the number of sub-blocks increases. On the other hand, when not encoding the quantization parameters of the divided sub-blocks, the load of the encoding process decreases, but the image quality deteriorates. In other words, in the conventional technology, when the number of sub-blocks increases due to division, it has been difficult to reduce the load of the encoding process while suppressing the deterioration of the image quality.

[0007] Therefore, this disclosure provides a technology that can reduce the load on encoding processing while suppressing a decrease in image quality even when the number of subblocks is divided.

[0008] To solve this problem, for example, the image encoding device of the present disclosure has the following configuration: a first division means for dividing an image into a plurality of blocks, a second division means for dividing the blocks into a plurality of subblocks, a prediction means for calculating a prediction error based on a prediction image generated by performing a prediction process in units of the subblocks, a quantization means for quantizing the prediction error using quantization parameters to generate quantization coefficients, and an encoding means for encoding the quantization coefficients and the quantization parameters. When the prediction means divides a subblock into four subblocks, the quantization means determines whether or not to share the quantization parameters among the four subblocks based on a comparison between size information corresponding to the size of the four subblocks and a quantization control threshold indicating a unit for encoding the quantization parameters, and the encoding means switches the encoding process for the quantization parameters of the four subblocks based on the determination.

[0009] According to this disclosure, even if the number of subblocks increases due to division, the load on the encoding process can be reduced while suppressing a decrease in image quality.

[0010] Other features and advantages of the technical ideas derived from this disclosure will become apparent from the following description with reference to the attached drawings. In the attached drawings, the same or similar components are given the same reference numeral.

[0011] The attached drawings are included in the specification and constitute part thereof, illustrating embodiments in this disclosure and used together with their descriptions to explain the technical ideas derived from this disclosure. Block diagram showing the configuration of the image encoding device of the first embodiment. Block diagram showing the configuration of the image decoding device of the second embodiment. Flowchart showing the encoding process in the image encoding device of the first embodiment. Flowchart showing the decoding process in the image decoding device of the second embodiment. Block diagram showing an example of a computer hardware configuration applicable to the image encoding device and decoding device of the embodiment. Diagram showing an example of the structure of a bitstream of the embodiment. Diagram showing an example of the structure of a bitstream of the embodiment. Diagram showing an example of a basic block by the prediction unit of the embodiment. Diagram showing an example of the type of division of a basic block by the prediction unit of the embodiment. Diagram showing an example of the type of division of a basic block by the prediction unit of the embodiment. Diagram showing an example of the type of division of a basic block by the prediction unit of the embodiment. Diagram showing an example of the type of division of a basic block by the prediction unit of the embodiment. Diagram showing an example of the type of division of a basic block by the prediction unit of the embodiment. Diagram showing an undivided basic block in the embodiment. Diagram showing the relationship between the type of subblock division and the size of the subblock in the embodiment. Diagram showing the relationship between the type of subblock division and the size of the subblock in the embodiment. A diagram showing the relationship between the type of subblock division and the size of the subblock in the embodiment. A diagram showing the relationship between the type of subblock division and the size of the subblock in the embodiment. A diagram showing the relationship between the type of subblock division and the size of the subblock in the embodiment. A diagram illustrating the method of dividing the basic block and the method of encoding and decoding the quantization parameters in the embodiment. A diagram illustrating the method of dividing the basic block and the method of encoding and decoding the quantization parameters in the embodiment. A diagram illustrating the method of dividing the basic block and the method of encoding and decoding the quantization parameters in the embodiment. A diagram illustrating the method of dividing the basic block and the method of encoding and decoding the quantization parameters in the embodiment. A diagram illustrating the method of dividing the basic block and the method of encoding and decoding the quantization parameters in the embodiment. A diagram illustrating the method of dividing the basic block and the method of encoding and decoding the quantization parameters in the embodiment.A diagram illustrating the case where quantization parameters are encoded in two of the four subblocks in the embodiment. A diagram illustrating the case where quantization parameters are encoded in two of the four subblocks in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram showing the relationship between the encoding of quantization parameters and the significance coefficient in each of the four subblocks divided into a quadtree in the embodiment. A diagram explaining the case in the embodiment where there is no significance coefficient in the unit that shares the quantization parameters. A diagram explaining the case in the embodiment where there is no significance coefficient in the unit that shares the quantization parameters. A diagram explaining the case in the embodiment where the quantization parameters are encoded in each of the four subblocks divided into a quadtree in the embodiment. A diagram explaining the case in the embodiment where the quantization parameters are encoded in each of the four subblocks divided into a quadtree in the embodiment. A diagram explaining the case in the embodiment where the quantization parameters are encoded in each of the four subblocks divided into a quadtree in the embodiment.A diagram illustrating the case in which quantization parameters are encoded in each of the subblocks partitioned into a quadtree in the embodiment. A diagram illustrating a method for setting quantization parameters in a subblock when the first subblock in the encoding order does not contain the significance coefficient.

[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessary, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] (First Embodiment) Figure 1 is a block diagram showing the configuration of an image encoding device according to the first embodiment. The image encoding device divides a basic block obtained by dividing an image into a plurality of subblocks, encodes each subblock to generate a bitstream, and outputs it. The image encoding device has a control unit 100, a terminal 101, a block division unit 102, a generation unit 103, a prediction unit 104, a conversion / quantization unit 105, an inverse quantization / inverse conversion unit 106, an image playback unit 107, a frame memory 108, an in-loop filter unit 109, an encoding unit 110, an integrated encoding unit 111, and a terminal 112.

[0014] Some or all of the block division unit 102, generation unit 103, prediction unit 104, conversion / quantization unit 105, inverse quantization / inverse conversion unit 106, image playback unit 107, frame memory 108, in-loop filter unit 109, encoding unit 110, and integrated encoding unit 111 of the image encoding device may be implemented by one or more circuits such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0015] In Figure 1, the control unit 100 is a processor that controls the entire image encoding device.

[0016] Terminal 101 is an input terminal into which image data is input.

[0017] The block division unit 102 divides the input image into multiple basic blocks and outputs tile images of each basic block to a subsequent stage.

[0018] The generation unit 103 generates and outputs information that serves as a reference for encoding the quantization parameters. The quantization parameters are parameters used for quantizing the transformation coefficients obtained by the orthogonal transformation. In this embodiment, the information that serves as a reference for encoding the quantization parameters is information indicating parameters (hereinafter referred to as quantization control thresholds) that are generated based on the number of pixels in the subblock. There are no particular limitations on the method for generating the quantization control thresholds, but the user may input the quantization control thresholds, the quantization control thresholds may be calculated from the characteristics of the input image, or a quantization control threshold specified in advance may be used as an initial value.

[0019] The prediction unit 104 generates subblocks by dividing the basic block. The prediction unit 104 performs at least one of intra-prediction (intra-frame prediction) or inter-prediction (inter-frame prediction) for each generated subblock, and generates predicted image data. Furthermore, the prediction unit 104 calculates and outputs the prediction error from the input pixel values ​​(image data) and the predicted image data. The prediction unit 104 may also output information necessary for prediction, such as subblock division, prediction mode, and motion vector, along with the prediction error. Hereafter, this information necessary for prediction will be referred to as prediction information.

[0020] The transformation and quantization unit 105 performs an orthogonal transformation of the prediction error in units of subblocks, and then quantizes it using quantization parameters to obtain residual coefficients. The residual coefficients are an example of quantization coefficients. Note that the quantization coefficients include coefficients obtained by quantizing the prediction error without performing an orthogonal transformation in units of subblocks.

[0021] The transformation / quantization unit 105 may determine the unit of subblocks in which quantization parameters are shared. For example, the transformation / quantization unit 105 may determine whether or not to share quantization parameters in subblocks based on a comparison between the number of pixels corresponding to the size of the subblock and a quantization control threshold. For example, the transformation / quantization unit 105 may determine whether or not to share quantization parameters in four subblocks. The four subblocks may be, for example, four subblocks generated by further quadrubbing a square subblock, which is generated by binary or ternary tree division of a rectangular subblock. Specifically, the transformation / quantization unit 105 may decide to share quantization parameters in four subblocks if the number of pixels corresponding to the size of the subblock is less than the quantization control threshold. On the other hand, the transformation / quantization unit 105 may decide to encode the quantization parameters in each of the four subblocks if the number of pixels corresponding to the size of the subblock is greater than the quantization control threshold. In other words, the quantization control threshold can also be said to be a value that indicates the unit of subblocks that encode quantization parameters. Even if the conversion / quantization unit 105 decides to encode the quantization parameters individually, the quantization parameters may be shared among some or all of the subblocks based on the value of the residual coefficient, etc. If the number of pixels corresponding to the size of the subblock is equal to the quantization control threshold, the conversion / quantization unit 105 may decide either way, but in this embodiment, it decides to encode the quantization parameters individually. Depending on this decision, the conversion / quantization unit 105 may encode one quantization parameter to be shared, or it may encode the quantization parameters of each subblock.

[0022] The inverse quantization / inverse transformation unit 106 inversely quantizes the residual coefficients output from the transformation / quantization unit 105 to reconstruct the transformation coefficients, and then performs an inverse orthogonal transformation to reconstruct the prediction error.

[0023] The frame memory 108 is a memory that stores the regenerated image data.

[0024] The image playback unit 107 obtains predicted image data by appropriately referring to the frame memory 108 based on the prediction information output from the prediction unit 104, and generates playback image data from this and the input prediction error.

[0025] The in-loop filter unit 109 performs in-loop filtering on the regenerated image, such as deblocking filtering and sample adaptive offsetting.

[0026] The encoding unit 110 encodes the residual coefficients output from the transformation / quantization unit 105 and the prediction information output from the prediction unit 104 to generate encoded data. The encoding unit 110 encodes the quantization parameters. The encoding unit 110 may switch the encoding process of the quantization parameters based on the decision made by the transformation / quantization unit 105 whether or not the quantization parameters are shared among the subblocks. For example, if the transformation / quantization unit 105 does not decide that the rectangular subblock is divided into four subblocks by a quadtree partition, the encoding unit 110 may execute a process (first process) to determine whether or not to encode the quantization parameters in each subblock. On the other hand, if the transformation / quantization unit 105 decides that the rectangular subblock is divided into four subblocks by a quadtree partition, the encoding unit 110 may execute a process (second process) to encode the quantization parameters shared among the subblocks.

[0027] The integrated encoding unit 111 encodes the information regarding the quantization control threshold from the generation unit 103 to generate header code data. Furthermore, the integrated encoding unit 111 combines this with the code data output from the encoding unit 110 to form a bitstream.

[0028] Terminal 112 is an output terminal that outputs the bitstream generated by the integrated encoding unit 111 to the outside.

[0029] The image encoding operation in the above-described image encoding device is explained below. In this embodiment, the device is configured to input moving image data frame by frame, but it may also be configured to input still image data for one frame.

[0030] Image data for one frame input from terminal 101 is input to block division unit 102.

[0031] The block division unit 102 divides the input image data into multiple basic blocks and outputs tile images of each basic block to the prediction unit 104.

[0032] The prediction unit 104 performs prediction processing on the tile image data input from the block division unit 102. Specifically, first, the prediction unit 104 decides on subblock division, which further divides the basic block into smaller subblocks.

[0033] Figures 7A to 7F show examples of the types of division of the basic block by the prediction unit 104 of the embodiment. The frame with a thick line represents the basic block 700. For simplicity of explanation, the size of the basic block 700 is assumed to be 32 x 32 pixels. Each rectangle separated by a thin line within the basic block 700 shown by the frame with a thick line represents a subblock. Figure 7A shows the basic block. Figure 7B shows an example of a square subblock obtained by dividing the basic block 700. In Figure 7B, the 32 x 32 pixel basic block is divided into a 16 x 16 pixel subblock. Figures 7C to 7F show multiple types of rectangular subblocks obtained by dividing the basic block 700. Figure 7C shows a vertically elongated 16 x 32 pixel subblock obtained by dividing the basic block 700. Figure 7D shows a horizontally elongated rectangular subblock of 32 x 16 pixels obtained by dividing the basic block 700. Figures 7E and 7F show rectangular subblocks obtained by dividing the basic block 700 in a ratio of 1:2:1.

[0034] The four subblocks in Figure 7B may be generated from the base block 700 in a single partition using a quadruple tree partition, or they may be generated from the base block 700 in two partitions using a binary tree partition. For example, the subblocks in Figure 7B are generated by dividing the base block 700 horizontally in a 1:1 ratio into two subblocks, and then further dividing those two subblocks vertically in a 1:1 ratio.

[0035] Thus, this embodiment performs encoding using not only square subblocks but also rectangular subblocks. Furthermore, this embodiment encodes information regarding the division type of such basic blocks as division information. Moreover, in order to obtain the hierarchical structure of subblocks as shown in the left-hand diagrams of Figures 9A to 9F described later, this embodiment encodes the division type information in a hierarchical manner.

[0036] The prediction unit 104 then determines a prediction mode for each subblock to be processed. Specifically, the prediction unit 104 determines a prediction mode for each subblock, such as intra-prediction using pixels encoded in the same frame as the frame containing each subblock to be processed, or inter-prediction using pixels from different encoded frames. The prediction unit 104 then generates predicted image data from the determined prediction mode and the encoded pixels. The prediction unit 104 further generates a prediction error from the input image data and the predicted image data and outputs it to the conversion / quantization unit 105. The prediction unit 104 also outputs information such as subblock division and prediction mode as prediction information to the encoding unit 110 and the image playback unit 107.

[0037] Here, the conversion and quantization processes performed by the conversion and quantization unit 105 will be explained in more detail. The conversion and quantization unit 105 performs frequency conversion on the prediction error of the subblock that has been predicted by the prediction unit 104, and then performs quantization. Figures 8A to 8F show the relationship between the type of block division and the size of the subblock. The size of the subblock may be represented by the number of pixels of the subblock corresponding to the division ratio of the block to be divided. The size of the subblock may also be represented by an index that substitutes for the number of pixels of the subblock. The generation of the index that substitutes for the number of pixels will be described later. The number of pixels and index that indicate the size of the subblock are examples of size information corresponding to the size of the subblock.

[0038] The transformation / quantization unit 105 determines, based on a comparison between the size of the subblock to be processed and the quantization control threshold output from the generation unit 103, which subblocks will share the quantization parameters and which subblocks will encode in the subsequent encoding unit 110. That is, the transformation / quantization unit 105 determines whether or not the quantization parameters should be shared among multiple subblocks based on a comparison between the quantization control threshold and the number of pixels in each subblock. For example, the transformation / quantization unit 105 determines whether or not the quantization parameters should be shared among the four subblocks that have been divided into a quadtree based on a comparison between the quantization control threshold and the number of pixels in each subblock. The encoding of the quantization parameters will be described later. There are no particular limitations on how the values ​​of the quantization parameters used for quantization are determined. For example, the values ​​of the quantization parameters may be values ​​entered by the user, values ​​calculated from the characteristics of the input image, or values ​​specified in advance as initial values.

[0039] Next, we will explain how to determine the unit used to encode the quantization parameters.

[0040] The conversion and quantization unit 105 compares the size of the subblock to be processed with the quantization control threshold to determine in what unit the quantization parameters will be encoded, that is, in what unit the same quantization parameters will be used.

[0041] Here, the calculation of the subblock size will be explained using Figures 8A to 8F. Figures 8A to 8F are diagrams showing the relationship between the type of subblock division and the subblock size in the embodiment. The size of the basic block, indicated by the thick outer frame in Figures 8A to 8F, is 32 x 32 pixels.

[0042] Here, the size of a subblock is represented by the number of pixels in the subblock corresponding to the division ratio of the block being divided. The numbers written in each subblock shown in Figures 8A to 8F represent the number of pixels in each subblock. In Figure 8A, the basic block is not divided. Therefore, the number of pixels in the subblock is the same as the basic block, 1024 (32 x 32) pixels. In Figure 8B, the basic block is divided into a quadtree. Therefore, the number of pixels in each subblock is 256 pixels. In Figures 8C and 8D, the basic block is divided into a binary tree. Therefore, the number of pixels in each subblock is 512 pixels. In Figures 8E and 8F, the basic block is divided into a ternary tree. Therefore, if the number of pixels in the subblock after division is halved, the subblock will have 512 pixels. Also, if the number of pixels in the subblock after division is quartile, the subblock will have 256 pixels.

[0043] Here, we will explain how the prediction unit 104 divides the basic block, and how the encoding unit 110 encodes the quantization parameters according to the quantization control threshold and the number of pixels in the subblock.

[0044] FIGS. 9A to 9F are diagrams for explaining a method of dividing a basic block and a method of encoding quantization parameters in an embodiment. In FIGS. 9A to 9F, a square indicated by the outermost thick frame indicates a basic block. In each of FIGS. 9A to 9F, the left diagram shows the division of sub-blocks, and the right diagram shows the area where quantization parameters are shared. Also, the numerical values in each block in the figure indicate the number of pixels of each sub-block, and in descending order of size, they are 128 pixels, 64 pixels, and 32 pixels. Qp indicates a quantization parameter. Four adjacent sub-blocks represented by the slanted lines from FIG. 9A to FIG. 9F represent four sub-blocks generated by a quadtree division. For example, on the left side of FIG. 9D, the four adjacent regions of the slanted 32-pixel sub-block indicate that the region before division is a 128-pixel rectangular region. More specifically, on the right side of FIG. 9D, the upper-left horizontally long rectangular region (128 pixels) surrounded by a thick line and the lower-left vertically long rectangular region (128 pixels) represent the rectangular region where the region before the above division is 128 pixels. Also, on the left side of FIG. 9D, the four adjacent regions of the slanted 64-pixel sub-block indicate that the region before division is a 256-pixel square region. More specifically, on the right side of FIG. 9D, the upper-right square region (256 pixels) and the lower-center square region (256 pixels) surrounded by a thick line represent the square region where the region before the above division is 256 pixels.

[0045] The division structure of each sub-block shown in FIGS. 9A to 9F is generated by a binary tree division, a ternary tree division, or a quadtree division dividing a basic block into a plurality of sub-blocks by four-stage recursive processing.

[0046] First, as the first-stage division, a vertical binary tree division divides a 1024-pixel basic block vertically. As a result, the basic block is divided into the division shape represented by the right side of FIG. 9B.

[0047] Next, as the second-stage division, the upper region (512 pixels) obtained by the first-stage horizontal binary tree division is divided into left and right parts. Also, the lower region (512 pixels) obtained by the first-stage horizontal ternary tree division is divided into three regions. As a result, the basic block is divided into the division shape shown on the right side of FIG. 9C.

[0048] Subsequently, as the third-stage division, the left region obtained by the second-stage division by binary tree division, that is, in the right side of FIG. 9C, the upper left square region (256 pixels) of the figure is divided vertically by binary tree division into upper and lower parts. Also, the right square region obtained by the second-stage division by binary tree division, that is, in the right side of FIG. 9C, the upper right square region (256 pixels) of the figure is divided into four square regions by quadtree division. As a result, the upper region (512 pixels) of the basic block is divided into six regions in the division shape represented by the upper region (512 pixels) on the right side of FIG. 9E. Further, among the three regions obtained by the second-stage division by ternary tree division, that is, in the right side of FIG. 9C, each of the left rectangular region (128 pixels) and the central square region (256 pixels) among the three lower regions of the figure is divided into four regions by quadtree division. As a result, the lower region (512 pixels) of the basic block is divided into nine regions in the division shape represented by the lower region (512 pixels) on the right side of FIG. 9F.

[0049] Finally, as the fourth-stage division, the upper rectangular region obtained by the third-stage binary tree division, that is, in the right side of FIG. 9D, the upper left rectangular region (128 pixels) of the figure is divided into four rectangular regions by quadtree division. As described above, by the recursive block division including quadtree division for square sub-blocks and rectangular sub-blocks, the division structure of the sub-blocks shown on the left side of FIGS. 9A to 9F is obtained.

[0050] Each of FIGS. 9A, 9B, 9C, 9D, 9E, and 9F shows the case where the quantization control threshold is 1024 pixels, 512 pixels, 256 pixels, 128 pixels, 64 pixels, and 32 pixels.

[0051] In the case of Figure 9A, that is, when the quantization control threshold is 1024 pixels, the number of pixels in all subblocks will be less than the quantization control threshold. In this case, a common quantization parameter is used in all subblocks in the figure, and the quantization parameter is encoded in association with the subblock containing the first significance coefficient in the encoding order (hereinafter also referred to as the encoding order). In this case, only one quantization parameter is encoded.

[0052] The significance coefficient is the non-zero residual coefficient among the residual coefficients after transformation and quantization. In other words, the presence of a significance coefficient indicates that there is at least one non-zero residual coefficient in the subblock after transformation and quantization. The relationship between the presence or absence of a significance coefficient and the coding of the quantization parameters will be described later using Figures 11A to 11F and Figures 12A to 12G.

[0053] In the case of Figure 9B, that is, when the quantization control threshold is 512 pixels, the number of pixels in each subblock obtained in the first binary tree split is 512 pixels or more, so the quantization parameters are shared in units of the blocks on the right side of Figure 9B. Furthermore, in units of that block, one quantization parameter is encoded in association with the subblock that contains the significance coefficient first in the encoding order. In this case, two quantization parameters are encoded.

[0054] In the case of Figure 9C, that is, when the quantization control threshold is 256 pixels, the quantization parameters are shared within the right-hand block of Figure 9C. Furthermore, within that block, one quantization parameter is encoded in association with the subblock that contains the significance coefficient at the beginning of the encoding order.

[0055] Here, we will explain the encoding of quantization parameters in ternary tree partitioning. In Figure 9C, in the ternary tree partitioned subblock located at the bottom, the number of pixels differs between the two end subblocks and the central subblock. Specifically, the two subblocks, which are divided into quarters of the number of pixels, have 128 pixels each, while the central subblock, which is divided into half the number of pixels, has 256 pixels. On the other hand, the value of the quantization control threshold is 256 pixels. In this case, since the number of pixels in the subblock with the largest number of pixels among the ternary tree partitioned subblocks is greater than or equal to the quantization control threshold, the quantization parameters are encoded in each of the three subblocks. In this case, five quantization parameters are encoded.

[0056] In the case of Figure 9D, that is, when the quantization control threshold is 128 pixels, the quantization parameters are shared within the block on the right side of the figure, and one quantization parameter is encoded in association with the subblock containing the significance coefficient first in the encoding order within that unit. As a result, in Figure 9D, there are six quantization parameters that are encoded.

[0057] In the case of Figure 9E, that is, when the quantization control threshold is 64 pixels, the quantization parameters are shared within the blocks on the right side of the figure, and the quantization parameters are encoded in association with the subblock that contains the significance coefficient first in the encoding order within that unit. In this case, there are 12 quantization parameters to be encoded.

[0058] In the case of Figure 9F, that is, when the quantization control threshold is 32 pixels, the quantization parameters are shared within the blocks on the right side of the figure, and the quantization parameters are encoded in association with the subblock that contains the significance coefficient first in the encoding order within that unit. In this case, there are 18 quantization parameters to be encoded.

[0059] In this embodiment, the quantization control threshold is compared with the number of pixels, but it may also be compared with an index that substitutes for the number of pixels in a subblock. Specifically, when the number of pixels in a subblock is a power of 2 and the exponent is an integer, the quantization control threshold may be compared with the exponent. There is a relationship between the number of pixels in a subblock and the index corresponding to the exponent, where the index decreases by 1 when the number of pixels in a subblock is halved. For example, the index corresponding to a number of pixels of 512 is 9, and the index corresponding to a number of pixels of 256 is 8.

[0060] Furthermore, when the left side of Figure 9E is represented using powers of 2, the subblock pixel counts of 32, 64, and 128 correspond to indices 5, 6, and 7, respectively. In this case, the quantization control threshold is 6. In this case, the quantization parameters are shared by the block units on the right side of Figure 9E, and the quantization parameters are encoded in association with the subblock containing the significance coefficient first in the coding order within that unit. Thus, even when comparing the quantization control threshold with an index that substitutes the number of pixels in a subblock, the encoding of the quantization parameters can be controlled in the same way as when comparing the quantization control threshold with the number of pixels. The value representing the quantization control threshold is smaller than the index that substitutes the number of pixels, so the quantization control threshold can be represented with less coding, especially when encoding using Golomb coding.

[0061] Returning to Figure 1, the inverse quantization / inverse transformation unit 106 inversely quantizes the input residual coefficients to reconstruct the transformation coefficients, and then performs an inverse orthogonal transformation on the reconstructed transformation coefficients to reconstruct the prediction error, which is then output to the image reconstruction unit 107. In the inverse quantization process of each subblock, the same quantization parameters used in the transformation / quantization unit 105 are used.

[0062] The image playback unit 107 appropriately refers to the frame memory 108 based on the prediction information input from the prediction unit 104 and acquires a predicted image. Then, the image playback unit 107 reconstructs the playback image data from the acquired predicted image and the reconstructed prediction error input from the inverse quantization / inverse transform unit 106, inputs it into the frame memory 108, and stores it.

[0063] The in-loop filter unit 109 reads the regenerated image from the frame memory 108 and performs in-loop filtering, such as deblocking filtering. The in-loop filtering is performed based on the prediction mode of the prediction unit 104, the values ​​of the quantization parameters used by the transformation / quantization unit 105, the presence or absence of non-zero values ​​(hereinafter referred to as significance coefficients) in the quantized subblocks, or subblock division information. The in-loop filter unit 109 then inputs the filtered image back into the frame memory 108 and stores it again.

[0064] The encoding unit 110 entropy encodes the residual coefficients generated by the transformation / quantization unit 105 and the prediction information input from the prediction unit 104 in units of subblocks to generate coded data.

[0065] The entropy coding method is not specifically designated, but Golomb coding, arithmetic coding, Huffman coding, etc., can be used. The generated coded data is output to the integrated coding unit 111. In coding the quantization parameters that constitute the quantization information, an identifier indicating the difference between the quantization parameter of the subblock to be coded and the predicted value calculated using the quantization parameters of subblocks coded before the subblock is coded is coded. In this embodiment, the quantization parameter coded immediately before the subblock in coding order is used as the predicted value, and the difference between it and the quantization parameter of the subblock is calculated, but the predicted value of the quantization parameter is not limited to this. The predicted value may be the quantization parameter of a subblock adjacent to the left or above the subblock in question, or it may be a value calculated from the quantization parameters of multiple subblocks, such as an average value. Furthermore, if the subblock to be processed is the first subblock in coding order among the subblocks belonging to the first basic block in a basic block row, the quantization parameter of the subblock in the basic block directly above it may be used as the predicted value. This enables parallel processing on a basic block row basis. Note that the first basic block in a basic block row refers to the basic block that has a picture boundary or tile boundary on its left side.

[0066] Here, the process of encoding quantization parameters into each subblock based on the quantization control threshold will be further explained using Figures 11A to 11F and 12A to 12G. Figures 11A to 11F show the relationship between the encoding of quantization parameters and the significance coefficient in the embodiment. The left side of Figures 11A to 11F shows the type of block division and the quantization parameter (Qp) used in each subblock. Subblocks with diagonal lines indicate the subblock associated with the encoded quantization parameter. The thick-lined frame indicates the region where the quantization parameter, determined based on the quantization control threshold and the number of pixels in the subblock to be processed, is shared. For example, if the number of pixels in all four subblocks is less than the quantization control threshold, the four subblocks share the quantization parameter.

[0067] The central figures in Figures 11A to 11F show whether each subblock has a significance coefficient. As mentioned above, a significance coefficient is a non-zero coefficient among the residual coefficients after transformation and quantization. In other words, having a significance coefficient means that there is at least one non-zero residual coefficient in the subblock after transformation and quantization. Also, the right-hand figures in Figures 11A to 11F show the coding order with arrows.

[0068] In this embodiment, within a region where quantization parameters are shared, the quantization parameters are encoded in association with the subblock that first contains the significance coefficient in the encoding order. For example, in Figure 11B, the subblock that first contains the significance coefficient in the encoding order is the upper right subblock, so the quantization parameters are encoded in association with that subblock. In this case, in the lower left and lower right subblocks, the quantization parameters of the subblocks in the encoding unit of the quantization parameters have already been encoded in the upper right subblock, so the quantization parameters are not encoded. On the other hand, in the quantization and dequantization processes in the lower left and lower right subblocks, QpA, which is the same quantization parameter as in the upper right subblock, is used. Furthermore, since there is no significance coefficient in the upper left subblock, the dequantization process is not performed, but QpA, which is the same quantization parameter as in the upper right subblock, is used for processes that use quantization parameters, such as deblocking filters.

[0069] Furthermore, in Figure 11F, the subblock containing the significance coefficient first in the encoding order is the lower subblock; therefore, the quantization parameter is encoded in association with that subblock, while the quantization parameter is not encoded for the upper and middle subblocks. However, in the upper and middle subblocks of Figure 11F, similar to the upper left subblock in Figure 11B, the same quantization parameter QpA as in the lower subblock is used for processes that utilize quantization parameters, such as deblocking filters.

[0070] Figures 12A to 12G show the relationship between the encoding of quantization parameters and the significance coefficient in each of the quadtree-divided subblocks in the embodiment. Next, using Figures 12A to 12G, we will explain the case in which quantization parameters are encoded in two subblocks in a quadtree-divided subblock. The meaning of the diagonally lined subblocks and the thick-lined frames in Figures 12A to 12G is the same as in Figures 11A to 11F, so we will omit the explanation. The right-hand figures in Figures 12A to 12G show the encoding order of the four subblocks generated by the quadtree division. The right-hand figures in Figures 12A to 12E show that the quantization parameters of each subblock are encoded in the order of the top-left subblock, top-right subblock, bottom-left subblock, and bottom-right subblock, while the right-hand figures in Figures 12F and 12G show that the quantization parameters of each subblock are encoded in the order of the top-left subblock, bottom-left subblock, top-right subblock, and bottom-right subblock.

[0071] Figures 12A, 12B, and 12C show the case where the top-left and top-right subblocks, and the bottom-left and bottom-right subblocks, share quantization parameters in the coding order of each subblock in a quadtree. Figures 12D, 12E, and 12F show the case where the top-left and bottom-left subblocks, and the top-right and bottom-right subblocks, share quantization parameters in the coding order of each subblock in a quadtree. Figure 12G will be discussed later.

[0072] In Figures 12A, 12B, and 12C, as explained in Figures 11A to 11F, the quantization parameter is encoded in association with the subblock that first contains the significance coefficient in the coding order. For example, in Figure 12C, the upper left and upper right subblocks share the quantization parameter. That is, the same quantization parameter QpA is used, but the subblock that first contains the significance coefficient in the coding order is the upper right subblock, so the quantization parameter is encoded in association with that subblock. Also, the lower left and lower right subblocks share the quantization parameter. That is, the same quantization parameter QpB is used, but the subblock that first contains the significance coefficient in the coding order is the lower left subblock, so the quantization parameter is encoded in association with that subblock. On the other hand, in Figures 12D and 12E, the quantization parameter is encoded in association with the subblock that first contains the significance coefficient among multiple subblocks that share the quantization parameter in the coding order. For example, in Figure 12E, the upper left and lower left subblocks share the quantization parameter. In other words, although the same quantization parameter QpA is used, of the upper left and lower left subblocks that share the quantization parameter in the coding order, the first subblock containing the significance coefficient is the upper left subblock, so the quantization parameter is encoded in association with that subblock. Also, the upper right and lower right subblocks share the quantization parameter. In other words, although the same quantization parameter QpB is used, of the upper right and lower right subblocks that share the quantization parameter in the coding order, the first subblock containing the significance coefficient is the lower right subblock, so the quantization parameter is encoded in association with that subblock.

[0073] Furthermore, in Figure 12F as well, the quantization parameters are encoded in association with the subblock that first contains the significance coefficient in the encoding order. In this case, the top-left and bottom-left subblocks share the quantization parameters. That is, the same quantization parameter QpA is used, but the subblock that first contains the significance coefficient in the encoding order is the bottom-left subblock, so the quantization parameters are encoded in association with that subblock. Also, the top-right and bottom-right subblocks share the quantization parameters. That is, the same quantization parameter QpB is used, but the subblock that first contains the significance coefficient in the encoding order is the top-right subblock, so the quantization parameters are encoded in association with that subblock. Note that there are no particular limitations on how to switch the encoding order of the four subblocks obtained by the quadtree partitioning, but a flag of 1 may be encoded to indicate that the quantization parameters are encoded in the encoding order shown in the right-hand diagram of Figure 12F, and a flag of 0 may be encoded to indicate that the quantization parameters are encoded in the encoding order shown in the right-hand diagrams of Figures 12A to 12E.

[0074] Thus, in subblocks within a region where quantization parameters determined by the quantization control threshold are shared, the quantization parameters are encoded in association with the subblock containing the significance coefficient first in the encoding order.

[0075] Next, using Figure 12G, we will explain the case where two regions sharing quantization parameters are set up in a quadtree-divided subblock, and the quantization parameters are encoded in one subblock. Figure 12G shows the case where the top-left and bottom-left subblocks, and the top-right and bottom-right subblocks, share quantization parameters in the encoding order of each subblock in the quadtree-divided subblock. In this case, the quantization parameters are encoded in association with the subblock that first contains the significance coefficient among the multiple subblocks that share the quantization parameters in the encoding order. For example, in Figure 12G, the top-left and bottom-left subblocks share quantization parameters. That is, the same quantization parameter QpA is used, but the top-left subblock is the first subblock to contain the significance coefficient in the encoding order, so the quantization parameters are encoded in association with that subblock.

[0076] Furthermore, in Figure 12G, the upper right and lower right subblocks share a quantization parameter. In this case, since the upper right and lower right subblocks that share a quantization parameter do not contain significance coefficients, the quantization parameter QpB is not encoded. However, for processes such as deblocking filters, the same value as the quantization parameter encoded immediately before, i.e., QpA, is used. Therefore, of the four subblocks, one quantization parameter is encoded in the upper left subblock.

[0077] Thus, compared to encoding the quantization parameters in two of the four subblocks obtained by the quadtree partitioning in the encoding order shown in the right-hand diagrams of Figures 12A to 12E, the quantization parameters of the four subblocks obtained by the quadtree partitioning can be represented with less coding. There are no particular limitations on how to switch the encoding order of the four subblocks obtained by the quadtree partitioning, but one flag may be encoded as 1 to indicate that the quantization parameters are encoded in the encoding order shown in the right-hand diagram of Figure 12G, and one flag may be encoded as 0 to indicate that the quantization parameters are encoded in the encoding order shown in the right-hand diagrams of Figures 12A to 12E. In this way, in the subblocks within the region where the quantization parameters determined by the quantization control threshold are shared, the quantization parameters are encoded in association with the subblock containing the significance coefficient first in the encoding order.

[0078] Figures 13A and 13B illustrate the case where no significance coefficient exists in the unit sharing quantization parameters in the embodiment. Here, the case where no significance coefficient is included in any subblock in the region where quantization parameters are shared will be further explained using Figures 13A and 13B. The left side of Figures 13A and 13B shows a subblock division similar to that in Figures 9A to 9F, and the numbers in the figure indicate the number of pixels in each subblock. The subblocks represented by the shaded lines in the figure represent the four subblocks divided by the quadtree division. The right side of Figures 13A and 13B is an example showing a shared region of quantization parameters determined based on the quantization control threshold and the number of pixels in each subblock, similar to Figures 9A to 9F and Figures 10A and 10B. Note that Figures 13A and 13B use the case where the quantization control threshold is 128 as an example. Figure 13A shows an example of encoding quantization parameters in each of the four subblocks when the number of pixels in at least one of the four subblocks is equal to or greater than the quantization control threshold. Figure 13B shows an example of encoding quantization parameters in two of the four subblocks. In this case, the sum of the number of pixels in the two subblocks is equal to or greater than the quantization control threshold.

[0079] In Figure 13A, we will explain the case where no significance coefficient exists in any of the four subblocks within the region corresponding to the quantization parameter QpE. In this case, the quantization parameter QpE for these four subblocks is not encoded. However, for processing such as deblocking filters, the same value as the quantization parameter encoded immediately before, i.e., QpD, is used.

[0080] As another example, in Figure 13B, if no significance coefficient exists in any of the two subblocks within the region corresponding to the quantization parameter QpF, the quantization parameter QpF for those two subblocks is not encoded. However, for processes such as deblocking filters, the same value as the quantization parameter encoded immediately before, i.e., QpE, is used.

[0081] In this embodiment, while it is stated that in the coding unit of a quantization parameter for which no significance coefficient exists, the quantization parameter used in the process of using the quantization parameter is the one coded immediately before in coding order, this embodiment is not limited to this. For example, in Figure 13A, QpB, which is the quantization parameter of the coding unit of the quantization parameter adjacent to the top, may be used as the quantization parameter QpE of the target subblock, or QpD, which is the quantization parameter adjacent to the left, may be used as the quantization parameter QpE. Furthermore, a value calculated from the quantization parameters of the coding unit of multiple quantization parameters, such as the mean value, may be used as the quantization parameter of the target subblock. For example, the average value of QpB and QpD may be used as the quantization parameter QpE of the target subblock. Moreover, initial values ​​of quantization parameters for tiles or tile groups composed of multiple tiles may be used. A tile is a unit into which a frame is divided, and is composed of at least one basic block.

[0082] In this embodiment, it has been explained that if the number of pixels in at least one of the four subblocks divided into a quadtree is equal to or greater than the quantization control threshold, the quantization parameters are encoded in each of the four subblocks. However, this is not limited to this. For example, the quantization parameters may be encoded in two of the four subblocks. Figures 10A and 10B illustrate the case in which the quantization parameters are encoded in two of the four subblocks in the embodiment. Figures 10A and 10B show the subblocks divided into a quadtree. The thick lines in Figures 10A and 10B indicate the units for encoding the quantization parameters. In other words, the subblocks within the thick lines share the quantization parameters.

[0083] Figure 10A shows the case where the quantization parameters are shared between the first (UL) and second (UR) subblocks and between the third (DL) and fourth (DR) subblocks in the coding order. Figure 10B shows the case where the quantization parameters are shared between the first (UL) and third (DL) subblocks and between the second (UR) and fourth (DR) subblocks in the coding order.

[0084] As shown in Figure 10A, when the quantization parameters are shared, the quantization parameters are encoded a total of two times: once in the first (UL) or second (UR) subblock (QpA) and once in the third (DL) or fourth (DR) subblock (QpB) in the encoding order. Also, as shown in Figure 10B, when the quantization parameters are shared, the quantization parameters are encoded a total of two times: once in the first (UL) or third (DL) subblock (QpA) and once in the second (UR) or fourth (DR) subblock (QpB) in the encoding order.

[0085] For example, in Figure 9D, the square region in the upper right (256 pixels) and the square region in the lower center (256 pixels) are each divided into four regions by a quadtree partition. If quantization parameters are encoded in two subblocks for each of the four regions obtained by the quadtree partition, then eight quantization parameters will be encoded, as shown in the right-hand diagram of Figure 13B. In this way, by encoding quantization parameters in two subblocks of the quadtree partitioned subblocks, it becomes possible to specify quantization parameters in finer units compared to encoding one quantization parameter in four subblocks.

[0086] The method for switching between using Figure 10A and Figure 10B for sharing quantization parameters is not particularly limited. For example, Figure 10A may be set as the default method, the user may specify it in advance, or it may be set in advance based on the presence or absence of edges such as object boundaries after analyzing the image. Furthermore, if the rectangular region formed by multiple subblocks that share quantization parameters to which the subblock encoded immediately before it belongs is a horizontal rectangle, Figure 10A may be used, and if it is a vertical rectangle, Figure 10B may be used. In this case, it is sufficient to refer to either the top or left information, and the decision of whether to use Figure 10A or Figure 10B can be made with minimal processing.

[0087] Alternatively, Figure 10A may be used when both the rectangular region formed by multiple subblocks sharing quantization parameters located at the top and the rectangular region formed by multiple subblocks sharing quantization parameters located on the left are horizontally elongated rectangles. Furthermore, Figure 10B may be used when both the rectangular region formed by multiple subblocks sharing quantization parameters located at the top and the rectangular region formed by multiple subblocks sharing quantization parameters located on the left are vertically elongated rectangles. In this case, the quantization parameters can be set adaptively by reflecting the information of the regions sharing quantization parameters located at the top and left, allowing for more appropriate control of image quality.

[0088] Figures 14A to 14F illustrate the case in which quantization parameters are encoded in each of the four subblocks that have been divided into a quadtree in the embodiment. For example, the transformation / quantization unit 105 performs quantization with the quantization parameters associated with the four subblocks if the number of pixels in any of the four subblocks, i.e., the maximum number of pixels, is greater than or equal to the quantization control threshold. The encoding unit 110 determines whether or not to encode the quantization parameters in each of the four subblocks if the number of pixels in any of the four subblocks, i.e., the maximum number of pixels, is greater than or equal to the quantization control threshold. For example, the encoding unit 110 determines whether or not to encode the quantization parameters based on the presence or absence of a significance coefficient in the subblock. Specifically, the encoding unit 110 may decide to encode the quantization parameters of a subblock that includes a significance coefficient. On the other hand, the encoding unit 110 may decide not to encode the quantization parameters of a subblock that does not include a significance coefficient. The meaning of the diagonally lined subblocks, the thick-lined frames, etc. in Figures 14A to 14F is the same as in Figures 11A to 11F, so an explanation is omitted.

[0089] In Figure 14A, all four subblocks contain significance coefficients. In this case, the quantization parameters are encoded in each of the four subblocks. That is, QpA is encoded in the first subblock, QpB in the second subblock, QpC in the third subblock, and QpD in the fourth subblock.

[0090] In Figure 14B, the first subblock in coding order does not contain the significance coefficient. In this case, the quantization parameter QpA is not coded in the first subblock. On the other hand, the second, third, and fourth subblocks in coding order contain the significance coefficient. Therefore, the quantization parameter QpB is coded in the second subblock, the quantization parameter QpC is coded in the third subblock, and furthermore, the quantization parameter QpD is coded in the fourth subblock located in the lower right.

[0091] Figure 15 illustrates a method for setting the quantization parameter in a subblock when the first subblock in the coding order does not contain a significance coefficient, according to the embodiment. In Figure 15, the meaning of the thick border and diagonal lines is the same as in Figures 14A to 14F. In Figure 15, the subblocks divided into four trees are the same as in Figure 14B. That is, the first subblock in the coding order does not contain a significance coefficient, while the second, third, and fourth subblocks do. As mentioned above, since the first subblock does not contain a significance coefficient, the quantization parameter QpA is not coded. In this case, the quantization parameter of the subblock located to the left or above the subblock, or the subblock immediately preceding it in the coding order, is set as QpA for the quantization parameter of the subblock. In Figure 15, Qpabov indicates the quantization parameter of the subblock located above, and Qpleft indicates the quantization parameter of the subblock located to the left. Furthermore, Qpprev indicates the quantization parameter of the subblock immediately preceding it in the coding order. Note that the squares labeled Qpabov, Qpleft, and Qppprev do not represent the shape of the subblock, but rather indicate the previous quantization parameter in top, left, and coding order.

[0092] In the case of Figure 14C, the second subblock in coding order does not contain the significance coefficient. In this case, the quantization parameter QpA is not encoded in the second subblock. At this time, the quantization parameter QpA of the first subblock, which is the preceding subblock in coding order, is set in the second subblock. On the other hand, in the case of Figure 14C, the first, third, and fourth subblocks in coding order contain the significance coefficient. Therefore, the quantization parameter QpA is encoded in the first subblock, QpB is encoded in the third subblock, and QpC is encoded in the fourth subblock.

[0093] In Figure 14D, the first and second subblocks in the coding order contain significance coefficients, but the third subblock does not. In this case, the quantization parameter QpB is not coded in the third subblock. At this time, the quantization parameter QpB of the second subblock, which is the preceding subblock in the coding order, is set in the third subblock. Furthermore, the quantization parameter QpC is coded in the fourth subblock.

[0094] In Figure 14E, the first, second, and third subblocks in coding order contain significance coefficients, but the fourth subblock does not. In this case, the quantization parameter QpC is not coded in the fourth subblock. At this time, the quantization parameter QpC of the third subblock, which is the immediately preceding subblock in coding order, is set in the fourth subblock.

[0095] Thus, as a result of comparing the quantization control threshold with the number of pixels in each subblock, when quantization parameters are encoded in all of multiple subblocks, the quantization parameters are encoded in association with the subblock containing the significance coefficient. If a subblock does not contain a significance coefficient, the quantization parameters are not encoded in that subblock, but the quantization parameters of the immediately preceding subblock in the encoding order are set for that subblock. However, the quantization parameters set for subblocks that do not contain a significance coefficient are not limited to these; they may also be the quantization parameters of the subblocks adjacent to the left and above, or the average value of the quantization parameters of multiple subblocks.

[0096] The integrated encoding unit 111 encodes information regarding the quantization control threshold. In this embodiment, the integrated encoding unit 111 encodes the number of pixels in the subblock as information regarding the quantization control threshold. The number of pixels in the subblock is a positive integer; for example, in the partitioning example shown in Figure 9B, 512 is encoded as the quantization control threshold. However, in this embodiment, the information regarding the quantization control threshold is not limited to the number of pixels in the subblock.

[0097] The integrated encoding unit 111 may encode an index calculated based on the number of pixels in the subblock. In this embodiment, when the number of pixels in the subblock can be expressed as a power of 2, the value of the exponent is encoded as the index. For example, when the number of pixels in the subblock is 512, the corresponding quantization control threshold is 512 = 2^9, so the numerical value representing the exponent of 2, 9, is encoded. Similarly, when the number of pixels in the subblock is 128, 128 = 2^7, so 7 is encoded.

[0098] On the other hand, if the ratio of each subblock to the number of pixels in the base block can be calculated based on the number of pixels in the base block, and the reciprocal of that value can be expressed as a power of 2, then the value of that power of 2 may be used as the index for encoding. For example, if the base block has 1024 (32 x 32) pixels and the quantization control threshold is 1024, then the ratio to the number of pixels in the base block is 1, and its reciprocal is also 1, so it can be expressed as 1 = 2^0. In this case, 0 is encoded. Alternatively, if the quantization control threshold is 64, then the ratio to the number of pixels in the base block is 1 / 16, and its reciprocal is 16, so it can be expressed as 16 = 2^4. In this case, 4 is encoded, and so on. By encoding 0 when the quantization control threshold is at its maximum value (the same as the number of pixels in the base block), the quantization control threshold can be encoded with fewer bits, especially when encoding using Golomb coding or similar methods.

[0099] The method for encoding information regarding the quantization control threshold is not specifically defined, but Golomb coding, arithmetic coding, Huffman coding, etc., can be used. The integrated coding unit 111 also multiplexes these codes and the coded data input from the coding unit 110 to form a bitstream. Finally, the integrated coding unit 111 outputs the bitstream to the outside from terminal 112.

[0100] Figures 6A and 6B show examples of bitstream structures. The bitstream contains encoded information about the quantization control threshold. This information is included in either a sequence or picture header. In this embodiment, the information about the quantization control threshold is included in the picture header, as shown in Figure 6A. However, the encoding location is not limited to this; the information about the quantization control threshold may also be included in the sequence header, as shown in Figure 6B.

[0101] Figure 3 is a flowchart showing the encoding process in the image encoding device of the first embodiment.

[0102] First, in the encoding process, in step S301, the block division unit 102 divides the input image in frame units into tile images in basic block units.

[0103] In step S302, the generation unit 103 determines a quantization control threshold that indicates the unit for encoding the quantization parameters. The generation unit 103 then uses the information of the quantization control threshold as quantization control threshold information. The quantization control threshold information is encoded by the integrated encoding unit 111.

[0104] In step S303, the prediction unit 104 performs a division process on the basic block image data generated in step S301 to generate subblocks. The prediction unit 104 then performs a prediction process on each of the generated subblocks to generate prediction information such as block division and prediction mode, as well as predicted image data. Furthermore, it calculates the prediction error from the input image data and the predicted image data.

[0105] In step S304, the transformation / quantization unit 105 performs an orthogonal transformation on the prediction error calculated in step S303 to generate transformation coefficients. Furthermore, the transformation / quantization unit 105 performs quantization using the quantization parameters determined based on the quantization control threshold information generated in step S302 to generate residual coefficients. Specifically, as described above, the transformation / quantization unit 105 compares the quantization control threshold information with the number of pixels in the subblock to determine whether or not to share quantization parameters among the subblocks within the basic block. Based on this determination, the transformation / quantization unit 105 performs quantization of each subblock using the quantization parameters corresponding to the subblocks in each region to generate residual coefficients for each subblock.

[0106] In step S305, the inverse quantization / inverse transformation unit 106 inversely quantizes and inversely orthogonal transforms the residual coefficients generated in step S304 to reconstruct the prediction error. The same quantization parameters used in step S304 are used for the inverse quantization process in this step.

[0107] In step S306, the image playback unit 107 acquires a predicted image based on the prediction information generated in step S303. Furthermore, the image playback unit 107 reconstructs the playback image data from the acquired predicted image and the prediction error generated in step S305.

[0108] In step S307, the encoding unit 110 encodes the block division information along with the prediction information generated in step S303 and the residual coefficients generated in step S304 to generate encoded data. The encoding unit 110 also encodes the quantization parameters used in step S304 based on the quantization control threshold information generated in step S302. Furthermore, the integrated encoding unit 111 generates a bitstream including other encoded data. Specifically, in step S304, the encoding unit 110 encodes the quantization parameters in association with at least one significance coefficient in the subblocks to be encoded in the region where the determined quantization parameters are shared. The integrated encoding unit 111 generates a bitstream including the encoded quantization parameters.

[0109] In step S308, the control unit 100 of the image encoding device determines whether or not the encoding of all basic blocks in the frame has been completed. If it has been completed, it proceeds to step S309; ​​otherwise, it targets the next basic block and returns to step S303.

[0110] In step S309, the in-loop filter unit 109 performs in-loop filtering on the image data reproduced in step S306, generates a filtered image, and terminates the process.

[0111] As described above, the image encoding device of this embodiment determines whether or not to share quantization parameters among the four subblocks based on a comparison between the sizes of the four subblocks obtained by quadtree partitioning of a rectangular subblock and a quantization control threshold. Based on this determination, the image encoding device switches the encoding process for the quantization parameters of the four subblocks. As a result, even if a rectangular subblock is divided into four in a single quadtree partition, the image encoding device can appropriately determine whether or not to encode the quantization parameters in the four subblocks and encode them. Consequently, when the subblock size is small, the image encoding device can reduce the encoding load compared to encoding each subblock individually while suppressing the degradation of image quality compared to not encoding the quantization parameters by encoding them and sharing them among multiple subblocks. Furthermore, when the subblock size is large, the image encoding device can improve image quality by encoding the quantization parameters in each subblock.

[0112] Furthermore, the image encoding device can achieve the above effect by generating four subblocks in a single quadtree partition, without performing binary tree partitioning twice on a rectangular subblock. This reduces the subblock partitioning process, improves the encoding efficiency of quantization parameters, and enables high-quality and highly efficient compression.

[0113] In step S302, the image encoding device generates quantization control threshold information, and in steps S304 and S307, it performs quantization and encoding based on the quantization control threshold information. This enables the image encoding device to achieve encoding with appropriate quantization parameters. As a result, the image encoding device can improve the image quality of the encoded image while reducing the total amount of data in the generated bitstream.

[0114] For example, if the size of a subblock is greater than or equal to the quantization control threshold, the image encoding device determines whether or not to encode quantization parameters in each subblock. On the other hand, if the size of a subblock is less than the quantization control threshold, the image encoding device encodes the quantization parameters shared by the four subblocks. This allows the image encoding device to encode quantization parameters appropriately according to the size of the subblocks.

[0115] The image encoding device determines whether or not to encode quantization parameters in each subblock based on the presence or absence of significance coefficients. This allows the image encoding device to encode only the necessary quantization parameters and omit encoding unnecessary ones, thereby reducing the processing burden.

[0116] In this embodiment, image data is input frame by frame, and an encoding process is performed to generate and output a bitstream. However, the encoding process is not limited to image data. For example, feature data used in machine learning, such as object recognition, may be input in a two-dimensional shape, and an encoding process may be performed to encode the bitstream. This makes it possible to efficiently encode feature data used in machine learning.

[0117] (Second Embodiment) Figure 2 is a block diagram showing the configuration of the image decoding device of the second embodiment. In this embodiment, the decoding of encoded data generated in the image encoding device shown in Figure 1 will be explained as an example. The image decoding device, for example, decodes a bitstream generated by encoding each of several subblocks into which the basic blocks into which the image has been divided, and then reproduces the image. The image decoding device includes a terminal 201, a separation decoding unit 202, a decoding unit 203, an inverse quantization / inverse transformation unit 204, an image reproduction unit 205, a frame memory 206, an in-loop filter unit 207, a terminal 208, and a control unit 200.

[0118] Some or all of the image decoding device's separation and decoding unit 202, decoding unit 203, inverse quantization and inverse transformation unit 204, image playback unit 205, frame memory 206, and in-loop filter unit 207 may be implemented by one or more circuits such as an ASIC and a PLD including an FPGA.

[0119] Terminal 201 is an input terminal to which the encoded bitstream is input.

[0120] The separation and decoding unit 202 separates the bitstream into coded data related to the decoding process and residual coefficients, and decodes the coded data present in the header portion of the bitstream. In this embodiment, the separation and decoding unit 202 decodes the quantization control threshold information as coded data and outputs it to the subsequent decoding unit 203 and inverse quantization / inverse transformation unit 204. The separation and decoding unit 202 operates in almost the opposite way to the integrated encoding unit 111 in Figure 1.

[0121] The decoding unit 203 decodes and obtains block division information, residual coefficients, prediction information, and quantization parameters from the coded data output from the separation decoding unit 202.

[0122] The decoding unit 203 may switch the decoding process for the quantization parameters of a subblock based on a comparison between the size of the subblock and the quantization control threshold. For example, if the multiple subblocks to be decoded are four subblocks obtained by dividing a rectangular subblock into four subblocks, and the size of any of the four subblocks is greater than or equal to the quantization control threshold, the decoding unit 203 may perform a process (third process) to determine whether or not to decode the quantization parameters for each of the four subblocks. On the other hand, if the multiple subblocks to be decoded are four subblocks obtained by dividing a rectangular subblock into four subblocks, and the size of all four subblocks is less than the quantization control threshold, the decoding unit 203 may perform a process (fourth process) to decode the quantization parameters shared by the four subblocks. The size of the subblock here may be the number of pixels in the subblock.

[0123] The inverse quantization / inverse transformation unit 204 performs inverse quantization on the residual coefficients input in block units, and then performs an inverse orthogonal transformation to obtain the prediction error.

[0124] The frame memory 206 is a memory that stores the data of the reproduced image (also called a picture).

[0125] The image playback unit 205 obtains predicted image data by appropriately referring to the frame memory 206 based on the input prediction information. Then, the image playback unit 205 generates and outputs reproduced image data from the predicted image data and the prediction error reproduced by the inverse quantization / inverse transformation unit 204.

[0126] The in-loop filter unit 207, similar to the in-loop filter unit 109 in Figure 1, performs in-loop filtering, such as deblocking filtering, on the regenerated image and outputs the filtered image.

[0127] Terminal 208 is an output terminal that outputs the reproduced image data to an external device.

[0128] In Figure 2, the control unit 200 is a processor that controls the entire image decoding device.

[0129] The image decoding operation in the above-described image decoding device is explained below. In this embodiment, the bitstream encoded in the first embodiment is decoded.

[0130] The separation and decoding unit 202 acquires the bitstream input from terminal 201. The separation and decoding unit 202 separates the coded data related to the decoding process and coefficients from the bitstream and decodes the coded data present in the header portion of the bitstream. Specifically, the separation and decoding unit 202 decodes the quantization control threshold information. In this embodiment, first, the quantization control threshold information is decoded from the picture header of the bitstream shown in Figure 6A. The quantization control threshold information obtained in this way is output to the decoding unit 203 and the inverse quantization / inverse transformation unit 204. Furthermore, the separation and decoding unit 202 outputs the coded data in block units of the picture data to the decoding unit 203.

[0131] The decoding unit 203 decodes the coded data and obtains block division information, residual coefficients, prediction information, and quantization parameters. Based on the block division information, the decoding unit 203 determines the division shape and number of pixels of the subblock to be processed. The decoding unit 203 outputs the residual coefficients and quantization parameters to the inverse quantization / inverse transformation unit 204. The decoding unit 203 outputs the obtained prediction information to the image playback unit 205. The decoding unit 203 decodes the quantization parameters in association with the subblock. In decoding the quantization parameters that constitute the quantization information, the decoding unit 203 decodes an identifier that shows the difference between the quantization parameters of the subblock to be decoded and the predicted value calculated using the quantization parameters of subblocks decoded before that subblock.

[0132] In this embodiment, the quantization parameter of a subblock is calculated by adding a difference value to the quantization parameter of the subblock that was decoded immediately before the subblock in the decoding order, using the predicted value as the quantization parameter. However, the predicted value of the quantization parameter is not limited to this. The decoding unit 203 may use the quantization parameter of a subblock adjacent to the left or above the subblock in question as the predicted value. Alternatively, the decoding unit 203 may use a value calculated from the quantization parameters of multiple subblocks, such as an average value, as the predicted value. Furthermore, if the subblock to be processed is the first subblock in the decoding order among the subblocks belonging to the first basic block in a basic block row, the decoding unit 203 may use the quantization parameter of the subblock in the basic block directly above it as the predicted value. This enables parallel processing on a basic block row basis. The first basic block in a basic block row is, for example, a basic block that has a picture boundary or tile boundary on its left side.

[0133] Here, the calculation of subblock size will be explained using Figure 8. Here, the subblock size is the number of pixels of the subblock corresponding to the division ratio of the block to be divided. Figure 8 shows examples of subblock division types when a 32x32 pixel base block is divided into subblocks. The numbers written in each subblock shown in Figures 8A to 8F represent the number of pixels of each subblock. Figure 8A shows that the base block has never been divided and the number of pixels of the subblock is the same as the base block, 1024 (32x32) pixels. Figure 8B shows that the base block has been divided into a quadtree and the number of pixels of each subblock is 256 pixels. Figures 8C and 8D show that the base block has been divided into a binary tree and the number of pixels of each subblock is 512 pixels. Figures 8E and 8F show that the base block has been divided into a ternary tree. When the number of pixels of the subblock after division is halved, the number of pixels of each subblock is 512 pixels. On the other hand, if the number of pixels in the sub-blocks after division is one-quarter, then the number of pixels in each sub-block will be 256.

[0134] Here, using Figures 9A to 9F, we will explain how the decoding unit 203 decodes the quantization parameters according to the quantization control threshold and the number of pixels in the subblock. In Figures 9A to 9F, the outermost square represents the basic block. In each of Figures 9A to 9F, the left side shows the subblock division, and the right side shows the region where the quantization parameters are shared. The numbers within each block in the figures represent the number of pixels in each subblock, which are 128 pixels, 64 pixels, and 32 pixels in descending order of size. Qp represents the quantization parameter. The four adjacent subblocks represented by the shaded lines in Figures 9A to 9F represent the four subblocks generated by the quadtree division. For example, on the left side of Figure 9D, the four adjacent regions of 32-pixel subblocks with the shaded lines represent a rectangular region with 128 pixels before the division. More specifically, on the right side of Figure 9D, the horizontal rectangular area in the upper left (128 pixels) enclosed by a thick line and the vertical rectangular area in the lower left (128 pixels) represent the rectangular areas that were 128 pixels before the division. Also, on the left side of Figure 9D, the four adjacent areas of 64-pixel subblocks with diagonal lines represent the square areas that were 256 pixels before the division. More specifically, on the right side of Figure 9D, the square area in the upper right (256 pixels) enclosed by a thick line and the square area in the lower center (256 pixels) represent the square areas that were 256 pixels before the division.

[0135] The subblock partitioning structures shown in Figures 9A to 9F are generated by dividing a basic block into multiple subblocks through a four-stage recursive process, using binary, ternary, or quadary tree partitioning. The partition shape and number of pixels of each subblock in the recursive partitioning process are determined based on the partitioning information of the block decoded by the decoding unit 203.

[0136] First, as the initial stage of division, the vertical binary tree division indicated by the division information divides the basic block (1024 pixels) vertically. As a result, the basic block is divided as shown on the right side of Figure 9B.

[0137] Next, in the second stage of division, the horizontal binary tree division indicated by the division information divides the upper region (512 pixels) obtained in the first division into left and right sections. In addition, the horizontal ternary tree division indicated by the division information divides the lower region (512 pixels) obtained in the first division into three regions. As a result, the basic block is divided as shown on the right side of Figure 9C.

[0138] Next, as the third stage of division, in the left region obtained by the second division using binary tree division, i.e., the right side of Figure 9C, the upper left square region (256 pixels) of the figure is divided vertically by the binary tree division indicated by the division information. Also, in the right square region obtained by the second division using binary tree division, i.e., the right side of Figure 9C, the upper right square region (256 pixels) of the figure is divided into four square regions by the quadrutree division indicated by the division information. As a result, the upper region (512 pixels) of the basic block is divided into six regions in the right side of Figure 9E, as shown by the division shape of the upper region (512 pixels). Furthermore, of the three regions obtained by the second division using ternary tree division, i.e., in the right side of Figure 9C, of ​​the three lower regions of the figure, the rectangular region (128 pixels) located on the left and the square region (256 pixels) located in the center are each divided into four regions by the quadrutree division indicated by the division information. As a result, the lower (512 pixel) area of ​​the basic block is divided into nine regions on the right side of Figure 9F, as shown by the division shape represented by the lower (512 pixel) area.

[0139] Finally, as the fourth stage of partitioning, the upper rectangular region obtained from the third binary tree partitioning, that is, the rectangular region (128 pixels) located in the upper left of Figure 9D on the right side, is divided into four rectangular regions by the quadtree partitioning indicated by the partitioning information. As described above, the partitioning structure of the subblocks shown on the left side of Figures 9A to 9F is obtained by recursive block partitioning including quadtree partitioning of the square subblocks and rectangular subblocks.

[0140] Figures 9A, 9B, 9C, 9D, 9E, and 9F show the cases where the quantization control threshold is 1024 pixels, 512 pixels, 256 pixels, 128 pixels, 64 pixels, and 32 pixels, respectively.

[0141] In the case of Figure 9A, i.e., when the quantization control threshold is 1024 pixels, a common quantization parameter is used for all subblocks in the figure, and the quantization parameter is decoded in association with the subblock containing the first significance coefficient in the decoding order. In this case, only one quantization parameter is decoded.

[0142] Note that the significance coefficient may be any non-zero residual coefficient among the residual coefficients after transformation and quantization. In other words, the presence of a significance coefficient indicates that at least one non-zero residual coefficient exists in the subblock obtained by decoding the coded data. The relationship between the presence or absence of a significance coefficient and the decoding of the quantization parameters will be described later using Figures 11A to 11F and Figures 12A to 12G.

[0143] In the case of Figure 9B, that is, when the quantization control threshold is 512 pixels, the number of pixels in each subblock obtained in the first binary tree split is 512 pixels or more, and the quantization parameters are shared in units of blocks on the right side of Figure 9B. Furthermore, in units of that block, one quantization parameter is decoded in association with the subblock that contains the significance coefficient first in the decoding order. In this case, two quantization parameters are decoded.

[0144] In the case of Figure 9C, that is, when the quantization control threshold is 256 pixels, the quantization parameters are shared within the right-hand block unit of Figure 9C. Furthermore, within that block unit, one quantization parameter is decoded in association with the subblock containing the significance coefficient at the beginning of the decoding order. Now, let's explain the decoding of quantization parameters in a ternary tree partition. In Figure 9C, in the ternary tree partitioned subblock located at the bottom, the number of pixels in the two end subblocks and the central subblock are different. Specifically, the two subblocks that are divided into quarters of the number of pixels have 128 pixels each, while the central subblock, which is divided into half the number of pixels, has 256 pixels. On the other hand, the value of the quantization control threshold is 256 pixels. In this case, since the number of pixels in the subblock with the largest number of pixels among the ternary tree partitioned subblocks is greater than or equal to the quantization control threshold, the quantization parameters are decoded for each of the three subblocks. In this case, five quantization parameters are decoded.

[0145] In the case of Figure 9D, that is, when the quantization control threshold is 128 pixels, the quantization parameters are shared within the block on the right side of the figure, and one quantization parameter is decoded in association with the subblock containing the significance coefficient at the beginning of the decoding order within that unit. As a result, in Figure 9D, there are six quantization parameters that are decoded.

[0146] In the case of Figure 9E, that is, when the quantization control threshold is 64 pixels, the quantization parameters are shared within the block on the right side of the figure, and the quantization parameters are decoded in association with the subblock containing the significance coefficient at the beginning of the decoding order within that unit. In this case, 12 quantization parameters are decoded.

[0147] In the case of Figure 9F, that is, when the quantization control threshold is 32 pixels, the quantization parameters are shared within the right-hand block of the figure, and the quantization parameters are decoded in association with the subblock containing the significance coefficient at the beginning of the decoding order within that unit. In this case, 18 quantization parameters are decoded.

[0148] In this embodiment, the quantization control threshold is compared with the number of pixels, but it may also be compared with an index that substitutes for the number of pixels in a subblock. Specifically, when the number of pixels in a subblock is a power of 2 and the exponent of 2 is an integer, the quantization control threshold may be compared with the exponent of 2. There is a relationship between the number of pixels in a subblock and the index corresponding to the exponent of 2, where the index decreases by 1 when the number of pixels in the subblock is halved. For example, the index corresponding to 512 pixels is 9, and the index corresponding to 256 pixels is 8. Also, when the left side of Figure 9E is represented by an exponent of 2, the number of pixels in the subblocks, 32, 64, and 128, correspond to indices 5, 6, and 7, respectively, and the quantization control threshold is 6. In this case, the quantization parameters are shared in units of blocks on the right side of Figure 9E, and the quantization parameters are decoded in association with the subblock containing the significance coefficient first in the decoding order for that unit. In this way, even when comparing the quantization control threshold with an index that substitutes for the number of pixels in a subblock, the decoding of the quantization parameters can be controlled in the same way as when comparing the quantization control threshold with the number of pixels. The value representing the quantization control threshold is smaller than the number of pixels because the index that substitutes for the number of pixels is smaller. Therefore, especially when encoding using Golomb coding, it is possible to decode a bitstream in which the quantization control threshold is represented with less code.

[0149] Here, the process of decoding quantization parameters to each subblock based on the quantization control threshold will be explained using Figures 11A to 11F and 12A to 12G. As described above, the left-hand figures of Figures 11A to 11F show the type of block division and the quantization parameter (Qp) used during encoding in each subblock. Subblocks with diagonal lines indicate subblocks to which quantization parameters are associated. The thick-lined boxes indicate regions where quantization parameters determined based on the quantization control threshold and the number of pixels in the subblocks to be processed are shared. For example, if the number of pixels in all four subblocks is less than the quantization control threshold, the four subblocks share the quantization parameter. The method of comparison with the quantization control threshold is the same as in the image encoding apparatus of the first embodiment.

[0150] The central figures in Figures 11A to 11F show whether each subblock has a significance coefficient. A significance coefficient is a non-zero residual coefficient. In other words, "having a significance coefficient" means that there is at least one non-zero residual coefficient within the subblock. The right-hand figures in Figures 11A to 11F show the decoding order with arrows.

[0151] In this embodiment, within a subblock of a quantization parameter coding unit, the quantization parameter is decoded in the subblock that first contains the significance coefficient in the decoding order. For example, in Figure 11B, the first subblock to contain the significance coefficient in the decoding order is the upper right subblock, so the quantization parameter is decoded for that subblock. In this case, the lower left and lower right subblocks are within a shared region where the quantization parameter is already decoded in the upper right subblock. That is, the encoded data for the quantization parameter corresponding to the lower left and lower right subblocks does not exist in the bitstream, and the quantization parameter corresponding to the lower left and lower right subblocks is not decoded. In the quantization and dequantization processes in the lower left and lower right subblocks, QpA, which is the same quantization parameter as the upper right subblock, is used. Furthermore, since there is no significance coefficient in the upper left subblock, the dequantization process is not performed, but QpA, which is the same quantization parameter as the upper right subblock, is used for processes that use the quantization parameter, such as the deblocking filter.

[0152] In Figure 11F, the subblock containing the significance coefficient is the lower subblock, which is the first to contain the significance coefficient in the decoding order. Therefore, the quantization parameter associated with that subblock is decoded. The encoded data for the quantization parameters corresponding to the upper and middle subblocks in Figure 11F does not exist in the bitstream, and the quantization parameters corresponding to the upper and middle subblocks are not decoded. However, in the upper and middle subblocks of Figure 11F, as with the upper left subblock in Figure 11B, the same quantization parameter QpA as in the lower subblock is used for processes that use quantization parameters, such as deblocking filters.

[0153] Next, using Figures 12A to 12G, we will explain the case where quantization parameters are decoded in two subblocks in a quadtree-partitioned subblock. The meaning of the shaded subblocks and thick-lined frames in Figures 12A to 12G is the same as in Figures 11A to 11F, so we will omit the explanation. The right-hand diagrams in Figures 12A to 12G show the decoding order of the four subblocks generated by the quadtree partitioning. The right-hand diagrams in Figures 12A to 12E show that the quantization parameters of each subblock are decoded in the order of the top-left subblock, top-right subblock, bottom-left subblock, and bottom-right subblock, while the right-hand diagrams in Figures 12F and 12G show that the quantization parameters of each subblock are decoded in the order of the top-left subblock, bottom-left subblock, top-right subblock, and bottom-right subblock.

[0154] Figures 12A, 12B, and 12C show the case where the top-left and top-right subblocks, and the bottom-left and bottom-right subblocks, share quantization parameters in the decoding order of each subblock in a quadtree. Figures 12D, 12E, and 12F show the case where the top-left and bottom-left subblocks, and the top-right and bottom-right subblocks, share quantization parameters in the decoding order of each subblock in a quadtree. Figure 12G will be discussed later.

[0155] In Figures 12A, 12B, and 12C, as explained in Figures 11A to 11F, the quantization parameters are decoded in association with the subblock that first contains the significance coefficient in the decoding order. For example, in Figure 12C, the upper left and upper right subblocks share the same quantization parameter. That is, the same quantization parameter QpA is used, but the first subblock to contain the significance coefficient in the decoding order is the upper right subblock, so the quantization parameter is decoded in association with that subblock. Similarly, the lower left and lower right subblocks share the same quantization parameter. That is, the same quantization parameter QpB is used, but the first subblock to contain the significance coefficient in the decoding order is the lower left subblock, so the quantization parameter is decoded in association with that subblock.

[0156] On the other hand, in Figures 12D and 12E, the quantization parameter is decoded in association with the subblock that first contains the significance coefficient among the multiple subblocks that share the quantization parameter in the decoding order. For example, in Figure 12E, the upper left and lower left subblocks share the quantization parameter. That is, the same quantization parameter QpA is used, but among the upper left and lower left subblocks that share the quantization parameter in the decoding order, the first subblock to contain the significance coefficient is the upper left subblock, so the quantization parameter is decoded in association with that subblock. Also, the upper right and lower right subblocks share the quantization parameter. That is, the same quantization parameter QpB is used, but among the upper right and lower right subblocks that share the quantization parameter in the decoding order, the first subblock to contain the significance coefficient is the lower right subblock, so the quantization parameter is decoded in association with that subblock.

[0157] Furthermore, in Figure 12F as well, the quantization parameters are decoded in association with the subblock that first contains the significance coefficient in the decoding order. In this case, the top-left and bottom-left subblocks share the quantization parameters. That is, the same quantization parameter QpA is used, but the subblock that first contains the significance coefficient in the decoding order is the bottom-left subblock, so the quantization parameters are decoded in association with that subblock. Also, the top-right and bottom-right subblocks share the quantization parameters. That is, the same quantization parameter QpB is used, but the subblock that first contains the significance coefficient in the decoding order is the top-right subblock, so the quantization parameters are decoded in association with that subblock. Note that there are no particular limitations on how to switch the decoding order of the four subblocks obtained by the quadtree partitioning, but a flag of 1 may be decoded to indicate that the quantization parameters are decoded in the decoding order shown in the right-hand diagram of Figure 12F, and a flag of 0 may be decoded to indicate that the quantization parameters are decoded in the decoding order shown in the right-hand diagrams of Figures 12A to 12E.

[0158] Thus, in the subblocks of the coding unit of the quantization parameters determined by the quantization control threshold, the quantization parameters are decoded for the subblock containing the significance coefficient first in the decoding order.

[0159] Next, using Figure 12G, we will explain the case where two regions are set up in a quadtree-divided subblock that share quantization parameters, and the quantization parameters are decoded in one subblock. Figure 12G shows the case where the top-left and bottom-left subblocks, and the top-right and bottom-right subblocks share quantization parameters in the decoding order of each subblock in the quadtree-divided subblock. In this case, the quantization parameters are decoded in association with the subblock that first contains the significance coefficient among the multiple subblocks that share the quantization parameters in the decoding order. For example, in Figure 12G, the top-left and bottom-left subblocks share quantization parameters. That is, the same quantization parameter QpA is used, but the top-left subblock is the first subblock to contain the significance coefficient in the decoding order, so the quantization parameters are decoded in association with that subblock.

[0160] Furthermore, in Figure 12G, the upper right and lower right subblocks share the same quantization parameter. In this case, since the upper right and lower right subblocks that share the quantization parameter do not contain the significance coefficient, the quantization parameter QpB is not decoded. However, for processes such as deblocking filters, the same value as the quantization parameter decoded immediately before, i.e., QpA, is used. Therefore, of the four subblocks, one quantization parameter is decoded in the upper left subblock. In this way, compared to decoding the quantization parameter in two of the four subblocks in the decoding order shown in the right-hand diagrams of Figures 12A to 12E, it is possible to decode a bitstream that represents the quantization parameter of the four subblocks with less coding power. The method for switching the decoding order of the four subblocks obtained by the quadtree partitioning is not particularly limited, but one option is to decode 1 as a flag indicating that the quantization parameters should be decoded in the decoding order shown in the right-hand diagram of Figure 12G, and decode 0 as a flag indicating that the quantization parameters should be decoded in the decoding order shown in the right-hand diagrams of Figures 12A to 12E. In this way, in the subblocks within the region where the quantization parameters determined by the quantization control threshold are shared, the quantization parameters are decoded in association with the subblock containing the significance coefficient first in the decoding order.

[0161] Here, we will further explain the case where no significance coefficients are included in any of the subblocks in the region where quantization parameters are shared, using Figures 13A and 13B. The left-hand figures of Figures 13A and 13B show subblock partitioning similar to that in Figures 9A to 9F, and the numbers in the figures indicate the number of pixels in each subblock. The subblocks represented by the shaded lines in the figures represent the four subblocks partitioned by the quadtree partitioning. The right-hand figures of Figures 13A and 13B are examples showing the shared region of quantization parameters determined based on the quantization control threshold and the number of pixels in each subblock, similar to Figures 9A to 9F and Figures 10A and 10B. Note that Figures 13A and 13B use the case where the quantization control threshold is 128 as an example. Figure 13A is an example of decoding the quantization parameters in each of the four subblocks when the number of pixels in at least one of the quadtree-partitioned subblocks is greater than or equal to the quantization control threshold. Figure 13B also shows an example of decoding the quantization parameters in two of the four subblocks.

[0162] At this time, the sum of the pixel counts of the two subblocks is greater than or equal to the quantization control threshold.

[0163] In Figure 13A, we will explain using the example where no significance coefficient exists in any of the four subblocks within the region corresponding to the quantization parameter QpE. In this case, the quantization parameter QpE for these four subblocks is not decoded. However, for processes such as deblocking filters, the same value as the quantization parameter decoded immediately before, i.e., QpD, is used. As another example, in Figure 13B, if no significance coefficient exists in any of the two subblocks within the region corresponding to the quantization parameter QpF, the quantization parameter QpF for these two subblocks is not decoded. However, for processes such as deblocking filters, the same value as the quantization parameter decoded immediately before, i.e., QpE, is used.

[0164] In this embodiment, while it is stated that in the coding unit of a quantization parameter for which no significance coefficient exists, the quantization parameter used in the processing that uses the quantization parameter is the one that was decoded immediately before in the decoding order, this embodiment is not limited to this. For example, in Figure 13A, as the quantization parameter QpE for which no significance coefficient exists, QpB, which is the quantization parameter of the coding unit of the quantization parameter adjacent to it above, may be used, or QpD, which is the quantization parameter adjacent to it to the left, may be used. Furthermore, as the quantization parameter for which no significance coefficient exists, a value calculated from the quantization parameters of the coding unit of multiple quantization parameters, such as the mean value, may be used. For example, the mean value of QpB and QpD may be used as the quantization parameter for which no significance coefficient exists. The initial value of the quantization parameter for a tile or a tile group composed of multiple tiles may be used as the quantization parameter for which no significance coefficient exists. Note that a tile is a unit that divides a frame and is composed of at least one basic block.

[0165] In this embodiment, it was explained that if the number of pixels in at least one of the four subblocks is equal to or greater than the quantization control threshold, the quantization parameters are decoded in each of the four subblocks. However, this is not limited to this. For example, the quantization parameters may be decoded in two of the four subblocks.

[0166] Figures 10A and 10B illustrate the case where quantization parameters are decoded in two of the four subblocks. Figures 10 and 10B show the subblocks of a quadtree. The thick-lined boxes indicate the units for decoding quantization parameters. Figure 10A shows the case where the 1st (UL) and 2nd (UR) subblocks, and the 3rd (DL) and 4th (DR) subblocks share quantization parameters in the decoding order. Figure 10B shows the case where the 1st (UL) and 3rd (DL) subblocks, and the 2nd (UR) and 4th (DR) subblocks share quantization parameters in the decoding order. When quantization parameters are shared as shown in Figure 10A, the quantization parameters are decoded a total of two times: once in the 1st (UL) or 2nd (UR) subblock (QpA), and once in the 3rd (DL) or 4th (DR) subblock (QpB). Furthermore, as shown in Figure 10B, if the quantization parameters are shared, the quantization parameters are decoded a total of two times: once in the first (UL) or third (DL) subblock (QpA) and once in the second (UR) or fourth (DR) subblock (QpB) in the decoding order.

[0167] For example, in Figure 9D, the square region in the upper right (256 pixels) and the square region in the lower center (256 pixels) are each divided into four regions by a quadtree partition. When the quantization parameters are decoded using two subblocks for each of the four regions obtained by the quadtree partition, eight quantization parameters are decoded, as shown in the right-hand diagram of Figure 13B. In this way, by decoded using two subblocks of the quadtree-partitioned subblocks, it becomes possible to decode a bitstream with quantization parameters specified in finer units compared to decoding one quantization parameter for each of the four subblocks.

[0168] The method for switching between using Figure 10A and Figure 10B for sharing quantization parameters is not particularly limited. For example, Figure 10A may be set as the default method, the user may specify it in advance, or it may be set in advance based on the presence or absence of edges such as object boundaries after analyzing the image. Furthermore, if the rectangular region formed by multiple subblocks that share quantization parameters to which the subblock decoded immediately before it belongs is a horizontal rectangle, Figure 10A may be used, and if it is a vertical rectangle, Figure 10B may be used. In this case, it is sufficient to refer to either the information from the top or the left, and the bitstream generated can be decoded with less processing power to decide whether to use Figure 10A or Figure 10B. Alternatively, if the rectangular region formed by multiple subblocks that share quantization parameters located on the top side and the rectangular region formed by multiple subblocks that share quantization parameters located on the left side are both horizontal rectangles, Figure 10A may be used. Furthermore, Figure 10B may be used when both the rectangular region formed by multiple subblocks sharing quantization parameters located at the top and the rectangular region formed by multiple subblocks sharing quantization parameters located on the left are vertically elongated rectangles. In this case, the quantization parameters can be set adaptively by reflecting the information of the regions sharing quantization parameters located at the top and left, making it possible to decode a bitstream with more appropriate image quality control.

[0169] Next, using Figures 14A to 14E, we will explain the case where quantization parameters are decoded in each subblock in a quadtree-divided subblock. For example, if the number of pixels in any of the four subblocks, i.e., the maximum number of pixels, is greater than or equal to the quantization control threshold, the quantization parameters are decoded in each subblock. The meaning of the diagonally lined subblocks and the thick-lined frames in Figures 14A to 14E is the same as in Figures 11A to 11F, so we will omit the explanation.

[0170] In Figure 14A, all four subblocks contain significance coefficients. In this case, the quantization parameters are decoded in each of the four subblocks. That is, QpA is decoded in the first subblock, QpB in the second subblock, QpC in the third subblock, and QpD in the fourth subblock.

[0171] In Figure 14B, the first subblock in the decoding order does not contain the significance coefficient. In this case, the quantization parameter QpA is not decoded in the first subblock. On the other hand, the second, third, and fourth subblocks in the decoding order contain the significance coefficient. Therefore, the quantization parameter QpB is decoded in the second subblock, the quantization parameter QpC is decoded in the third subblock, and furthermore, the quantization parameter QpD is decoded in the fourth subblock located in the lower right.

[0172] Here, we will explain, using Figure 15, how the quantization parameters are set for a subblock when the first subblock in the decoding order does not contain a significance coefficient. In Figure 15, the meaning of the thick border and diagonal lines is the same as in Figures 14A to 14E. In Figure 15, the subblocks divided into four trees are the same as in Figure 14B. That is, the first subblock in the decoding order does not contain a significance coefficient, while the second, third, and fourth subblocks do. As mentioned above, since the first subblock does not contain a significance coefficient, the quantization parameter QpA is not decoded. In this case, the quantization parameters Qpleft and Qpabov of the subblocks located to the left and above the subblock in question, or the quantization parameter Qppprev of the subblock immediately preceding the one in the decoding order, are set as the quantization parameters of the subblock in question. In Figure 15, Qpabov represents the quantization parameter of the subblock located above, and Qpleft represents the quantization parameter of the subblock located to the left. Furthermore, Qppprev indicates the quantization parameters of the immediately preceding subblock in the decoding order. Note that the squares labeled Qpabov, Qpleft, and Qppprev do not represent the shape of the subblock, but rather indicate the quantization parameters of the immediately preceding subblock in the top, left, and decoding order, respectively.

[0173] In the case of Figure 14C, the second subblock in the decoding order does not contain the significance coefficient. In this case, the quantization parameter QpA is not decoded in the second subblock. At this time, the quantization parameter QpA of the first subblock, which is the subblock immediately preceding it in the decoding order, is set in the second subblock. On the other hand, in the case of Figure 14C, the first, third, and fourth subblocks in the decoding order contain the significance coefficient. Therefore, the quantization parameter QpA is decoded in the first subblock, the quantization parameter QpB is decoded in the third subblock, and the quantization parameter QpC is decoded in the fourth subblock.

[0174] In Figure 14D, the first and second subblocks in the decoding order contain significance coefficients, but the third subblock does not. In this case, the quantization parameter QpB is not decoded in the third subblock. At this time, the quantization parameter QpB of the second subblock, which is the preceding subblock in the decoding order, is set in the third subblock. Furthermore, the quantization parameter QpC is decoded in the fourth subblock.

[0175] In Figure 14E, the first, second, and third subblocks in the decoding order contain significance coefficients, but the fourth subblock does not. In this case, the quantization parameter QpC is not decoded in the fourth subblock. At this time, the quantization parameter QpC of the third subblock, which is the immediately preceding subblock in the decoding order, is set in the fourth subblock.

[0176] Thus, as a result of comparing the quantization control threshold with the number of pixels in each subblock, when decoding quantization parameters in all of multiple subblocks, the quantization parameters are associated with and decoded in relation to the subblock containing the significance coefficient. If a subblock does not contain a significance coefficient, the quantization parameters are not decoded in that subblock, but the quantization parameters of the immediately preceding subblock in the decoding order are set for that subblock. However, the quantization parameters set for subblocks that do not contain a significance coefficient are not limited to these; they may also be the quantization parameters of the subblock adjacent to the left or above, or the average value of the quantization parameters of multiple subblocks.

[0177] The inverse quantization / inverse transformation unit 204 performs inverse quantization on the input residual coefficients to generate orthogonal transformation coefficients, and then performs an inverse orthogonal transformation to reconstruct the prediction error. In the inverse quantization of each subblock, the inverse quantization / inverse transformation unit 204 may use common quantization parameters for each region where quantization parameters are shared. The inverse quantization / inverse transformation unit 204 outputs the reconstructed prediction error to the image playback unit 205.

[0178] The image playback unit 205 appropriately references the frame memory 206 based on the prediction information input from the decoding unit 203 and acquires a predicted image. The image playback unit 205 generates playback image data from the acquired predicted image and the prediction error obtained from the inverse quantization / inverse transform unit 204. The image playback unit 205 outputs the playback image to the frame memory 206 and stores it. The stored playback image data is used as a reference during prediction.

[0179] The in-loop filter unit 207 reads the regenerated image from the frame memory 206, similar to the in-loop filter unit 109 in Figure 1, and performs in-loop filtering such as deblocking filtering and sample adaptive offsetting. The in-loop filter unit 207 then stores the filtered image back into the frame memory 206 as the filtered image.

[0180] The frame memory 206 ultimately outputs the filtered image stored by the in-loop filter unit 207 to the outside via terminal 208.

[0181] Figure 4 is a flowchart showing the decoding process in the image decoding device of the second embodiment.

[0182] First, in the decoding process, in step S401, the separation and decoding unit 202 separates the bitstream into coded data related to the decoding process and coefficients, decodes the coded data of the header portion, and obtains quantization control threshold information.

[0183] In this embodiment, the separation and decoding unit 202 decodes the number of pixels in the subblock as information regarding the quantization control threshold. The number of pixels in the subblock is a positive integer. For example, in the partitioning example shown in Figure 9B, the separation and decoding unit 202 decodes 512 as the quantization control threshold.

[0184] However, in this embodiment, the information regarding the quantization control threshold is not limited to the number of pixels in the subblock. The separation and decoding unit 202 may decode an index calculated based on the number of pixels in the subblock. In this embodiment, if the number of pixels in the subblock can be expressed as a power of 2, the separation and decoding unit 202 decodes the value of the exponent as the index. For example, if the number of pixels in the subblock is 512, the corresponding quantization control threshold is 512 = 2^9, so the separation and decoding unit 202 decodes 9, which is the numerical value representing the exponent of 2. Also, if the number of pixels in the subblock is 128, the separation and decoding unit 202 decodes 7, since 128 = 2^7.

[0185] On the other hand, if the ratio of each subblock to the number of pixels of the basic block is calculated based on the number of pixels of the basic block, and the reciprocal of that value can be expressed as a power of 2, the separation and decoding unit 202 may decode using the value of that power of 2 as the index. For example, if the basic block has 1024 (32 x 32) pixels and the quantization control threshold is 1024, the ratio to the number of pixels of the basic block is 1, and its reciprocal is also 1, so it can be expressed as 1 = 2^0. In this case, the separation and decoding unit 202 decodes 0. Alternatively, if the quantization control threshold is 64, the ratio to the number of pixels of the basic block is 1 / 16, and its reciprocal is 16, so it can be expressed as 16 = 2^4. In this case, the separation and decoding unit 202 decodes 4. Thus, when the quantization control threshold is at its maximum value (the same as the number of pixels of the basic block), the separation and decoding unit 202 decodes 0. This makes it possible to decode a bitstream that has encoded the quantization control threshold with fewer bits, especially when decoding a bitstream encoded using Golomb coding or similar methods.

[0186] In step S402, the decoding unit 203 decodes the coded data separated in step S401 and obtains block division information, residual coefficients, prediction information, and quantization parameters.

[0187] In step S403, the inverse quantization / inverse transformation unit 204 performs inverse quantization on the residual coefficients in sub-block units and then performs an inverse orthogonal transformation to obtain the prediction error. Specifically, the inverse quantization / inverse transformation unit 204 compares the quantization control threshold information with the size determined based on the number of pixels in the sub-blocks determined from the acquired block division information. As a result of the comparison, the inverse quantization / inverse transformation unit 204 determines the regions (sub-blocks) in which the quantization parameters are shared. Then, the inverse quantization / inverse transformation unit 204 performs inverse quantization processing based on the quantization parameters assigned to each sub-block.

[0188] In step S404, the image playback unit 205 acquires a predicted image based on the prediction information acquired in step S402. The image playback unit 205 generates playback image data from the acquired predicted image and the prediction error generated in step S403. The image playback unit 205 stores the generated playback image in the frame memory 206.

[0189] In step S405, the control unit 200 of the image decoding device determines whether decoding of all blocks in the frame has been completed. If it has been completed, it proceeds to step S406; otherwise, it returns to step S402, targeting the next block.

[0190] In step S406, the in-loop filter unit 207 performs in-loop filtering on the data of the regenerated image reproduced in step S404, and generates the filtered image as the filtered image. At this point, the image decoding device terminates its processing.

[0191] With the above configuration and operation, the image decoding device of this embodiment switches the decoding process for quantization parameters based on a comparison between the sizes of the four subblocks generated by quadtree partitioning of a rectangular subblock and the quantization control threshold. As a result, the image decoding device can decode quantization parameters even if the subblock to be decoded for quantization parameters is four subblocks generated by quadtree partitioning of a rectangular subblock.

[0192] The image decoding device can decode a bitstream with reduced data volume by decoding quantization parameters using quantization control threshold information.

[0193] The image decoding device determines whether or not to decode the quantization parameters for each of the four subblocks if the size of each subblock is greater than or equal to the quantization control threshold. On the other hand, if the size of all four subblocks is less than the quantization control threshold, the image decoding device decodes the quantization parameters shared by those four subblocks. This allows the image decoding device to appropriately decode the quantization parameters according to the size of the subblocks.

[0194] The image decoding device determines whether or not to decode the quantization parameters in each subblock based on the presence or absence of a significance coefficient. This allows the image decoding device to decode only the necessary quantization parameters and omit the decoding of unnecessary ones, thereby reducing the processing burden.

[0195] In the image decoding device, as shown in Figure 6A, the bitstream containing the quantization control threshold information in the picture header is decoded. However, the encoding location of the information is not limited to this. As shown in Figure 6B, it may be encoded in the sequence header of the image, or it may be encoded at any other location.

[0196] Furthermore, in this embodiment, image data is input frame by frame, and the bitstream generated by encoding is decoded. However, the target of the decoding process is not limited to the bitstream from which the image data has been encoded. For example, feature data used in machine learning, such as object recognition, may be input in a two-dimensional shape, and the bitstream generated by encoding is decoded. This makes it possible to decode a bitstream from which feature data used in machine learning has been efficiently encoded.

[0197] (Third Embodiment) In the above embodiment, each processing unit shown in Figures 1 and 2 was described as being composed of hardware. However, the configurations of the image encoding device and image decoding device shown in the above figures may be realized by the execution of a computer program (hereinafter also referred to as a program).

[0198] Figure 5 is a block diagram showing an example of the hardware configuration of a computer applicable to the image encoding device and image decoding device according to each embodiment. The computer has a CPU 501, RAM 502, ROM 503, operation unit 504, display unit 505, external storage device 506, I / F 507, and bus 508. The CPU 501, RAM 502, ROM 503, operation unit 504, display unit 505, external storage device 506, and I / F 507 are connected to each other by the bus 508 so that they can send and receive data.

[0199] CPU 501 is an abbreviation for Central Processing Unit and is a type of processor. CPU 501 controls the entire computer using computer programs and data stored in either RAM 502, ROM 503, or external storage device 506. CPU 501 executes each process performed by the image encoding device and image decoding device according to each embodiment described above. That is, CPU 501 functions as part or all of the configurations shown in Figures 1 and 2.

[0200] The computer may have other processors in place of, or in addition to, the CPU 501, such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), and a QPU (Quantum Processing Unit). Furthermore, the computer may have multiple processors of the same type, each performing a different function.

[0201] Some or all of the functions of the image encoding device and the image decoding device may be implemented by one or more processors, including the CPU 501, reading a program stored in an external storage device 506 or the like, loading it into the RAM 502, and executing it. Furthermore, some of the functions of the image encoding device and the image decoding device may be implemented by one or more circuits, such as an ASIC and a PLD including an FPGA.

[0202] RAM 502 stands for Random Access Memory and is a memory capable of high-speed data reading and writing. RAM 502 has an area for temporarily storing computer programs and data loaded from the external storage device 506, data acquired from external sources via the I / F 507, etc. Furthermore, RAM 502 has a work area used by the CPU 501 when executing various processes. That is, RAM 502 can be allocated as, for example, frame memory, or various other areas can be provided as appropriate.

[0203] ROM503 stands for Read Only Memory. ROM503 stores computer configuration data and boot programs, among other things.

[0204] The control unit 504 includes a keyboard, mouse, and the like. The control unit 504 receives user operations and instructions to the computer and outputs the received operations and instructions to the CPU 501.

[0205] The display unit 505 displays the processing results from the CPU 501. The display unit 505 is composed of, for example, a liquid crystal display.

[0206] The external storage device 506 is a large-capacity non-volatile information storage device, such as a hard disk drive or an SSD (Solid State Drive). The external storage device 506 stores the OS (operating system) and computer programs that enable the CPU 501 to implement the functions of each part shown in Figures 1 and 2. Furthermore, the external storage device 506 may also store image data to be processed. The computer programs and data stored in the external storage device 506 are loaded into the RAM 502 as appropriate according to the control of the CPU 501 and become the target of processing by the CPU 501.

[0207] I / F507 is an abbreviation for interface. I / F507 may be connected to networks such as LANs and the Internet, as well as other devices such as projection and display devices. Computers acquire and transmit various types of information via I / F507.

[0208] In the aforementioned computer, the CPU 501 primarily executes the operations described in the flowchart above.

[0209] (Other Embodiments) The configuration of the above-described embodiment may also be achieved by supplying a storage medium containing the code of a computer program that realizes the functions described above to a system, and by the system reading and executing the code of the computer program. In this case, the code of the computer program read from the storage medium itself realizes the functions of the above-described embodiment, and the storage medium containing the code of the computer program may constitute the above-described embodiment. Furthermore, it is also included in cases where an operating system (OS) or the like running on the computer performs some or all of the actual processing based on the instructions of the code of the program, and the above-described functions are realized by that processing.

[0210] Furthermore, the above-described embodiment may also be implemented in the following form: a processor or the like reads computer program code from a storage medium and writes it to the memory provided in a function expansion card inserted into a computer or a function expansion unit connected to a computer. The processor provided in the function expansion card or function expansion unit then performs some or all of the actual processing based on the instructions in the computer program code to realize the aforementioned function.

[0211] When the above-described embodiment is applied to the storage medium, the storage medium stores the code of a computer program corresponding to the flowchart described earlier.

[0212] The embodiments described above are applicable to encoding and decoding devices that encode and decode still images and videos. In particular, the embodiments described above are applicable to encoding and decoding methods that use quantization processing.

[0213] In the above embodiment, the four subblocks to be judged as to whether or not they share quantization parameters were square subblocks that were generated by quadruple-tree partitioning of square subblocks that were generated by partitioning (binary tree partitioning or ternary tree partitioning) of rectangular subblocks. However, the four subblocks to be judged are not limited to subblocks generated by the above partitioning.

[0214] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0215] The technical ideas derived from this disclosure are not limited to the exemplary embodiments disclosed, but are intended to encompass various modifications of the exemplary embodiments, or substitutions with equivalent structures or functions. The scope of the following claims should be interpreted in the broadest way to encompass all such modifications and equivalent structures and functions.

[0216] This application claims priority based on Japanese Patent Application No. 2025-027214, filed on 21 February 2025, and all of its contents are incorporated herein by reference.

[0217] 100... Control unit, 102... Block division unit, 104... Prediction unit, 105... Transformation / quantization unit, 110... Encoding unit, 202... Separation / decoding unit, 203... Decoding unit, 204... Inverse quantization / inverse transformation unit, 205... Image playback unit.

Claims

1. An image encoding device comprising: a first division means for dividing an image into a plurality of blocks; a second division means for dividing the blocks into a plurality of subblocks; a prediction means for calculating a prediction error based on a prediction image generated by performing a prediction process on the subblock units; a quantization means for quantizing the prediction error using quantization parameters to generate quantization coefficients; and an encoding means for encoding the quantization coefficients and the quantization parameters, wherein when the prediction means divides a subblock into four subblocks, the quantization means determines whether or not to share the quantization parameters among the four subblocks based on a comparison between size information corresponding to the size of the four subblocks and a quantization control threshold indicating a unit for encoding the quantization parameters; and the encoding means switches the encoding process for the quantization parameters of the four subblocks based on the determination.

2. If the size information of any of the four subblocks is greater than the quantization control threshold, the quantization means decides to encode a quantization parameter in each of the four subblocks, and the encoding means performs a first process to determine whether or not to encode a quantization parameter in each of the four subblocks, and if the size information of all four subblocks is less than the quantization control threshold, the quantization means decides to share a quantization parameter among the four subblocks, and the encoding means performs a second process to encode a quantization parameter shared among the four subblocks, characterized in that the image encoding apparatus according to claim 1.

3. The image encoding apparatus according to claim 2, characterized in that the encoding means encodes the quantization parameters of the subblock containing the significance coefficient among the four subblocks in the first processing.

4. The image encoding apparatus according to claim 2 or 3, characterized in that, in the first processing, the encoding means does not encode the quantization parameters of the subblocks among the four subblocks that do not contain significance coefficients.

5. The image encoding apparatus according to claim 4, characterized in that, in the first processing, the encoding means sets the quantization parameters of the subblock that does not include the significance coefficient using quantization parameters encoded before the subblock that does not include the significance coefficient.

6. The image encoding apparatus according to any one of claims 2 to 5, characterized in that, in the second processing, the encoding means encodes the quantization parameter of the subblock containing the significance coefficient first among the four subblocks in the encoding order, and sets the encoded quantization parameter as the quantization parameter of the other subblocks containing the significance coefficient among the four subblocks.

7. The image encoding apparatus according to any one of claims 1 to 6, characterized in that the prediction means divides the rectangular subblock into four subblocks by dividing the square subblock generated by dividing the rectangular subblock into four subblocks by quadtree division.

8. If the size information is greater than a quantization control threshold indicating a unit for encoding quantization parameters, the encoding means shares the quantization parameters encoded in horizontally adjacent subblocks if the plurality of subblocks that share the quantization parameters to which the immediately preceding encoded subblock belongs are horizontally elongated rectangles, and shares the quantization parameters encoded in vertically adjacent subblocks if the plurality of subblocks that share the quantization parameters to which the immediately preceding encoded subblock belongs are vertically elongated rectangles, characterized in that the image encoding apparatus according to claim 7.

9. An image decoding device that reconstructs an image by decoding a bitstream generated by encoding each of several subblocks obtained by dividing an image into several blocks, the device comprising: separation means for separating coded data from the bitstream; decoding means for decoding the quantization parameters of the subblocks from the coded data; inverse quantization means for inverse quantization of quantization coefficients based on the quantization parameters to generate a prediction error; and image reproduction means for reconstructing an image based on the prediction error, wherein the decoding means switches the decoding process for the quantization parameters of the four subblocks based on a comparison between size information corresponding to the size of the four subblocks and a quantization control threshold indicating the unit in which the quantization parameters are encoded, when the target of decoding the quantization parameters is four subblocks.

10. The image decoding apparatus according to claim 9, characterized in that the decoding means performs a third process to determine whether or not to decode the quantization parameters in each of the four subblocks if any of the size information of the four subblocks is greater than the quantization control threshold, and performs a fourth process to decode the quantization parameters shared by the four subblocks if all of the size information of the four subblocks is less than the quantization control threshold.

11. The image decoding apparatus according to claim 10, characterized in that the decoding means decodes the quantization parameters of the subblock containing the significance coefficient among the four subblocks in the third process.

12. The image decoding apparatus according to claim 10 or 11, characterized in that the decoding means does not decode the quantization parameters of the subblocks among the four subblocks that do not contain significance coefficients in the third process.

13. The image decoding apparatus according to claim 12, characterized in that, in the third process, the decoding means sets the quantization parameters of the subblock that does not contain the significance coefficient using the quantization parameters that were decoded before the subblock that does not contain the significance coefficient.

14. The image decoding apparatus according to any one of claims 10 to 13, characterized in that, in the fourth process, the decoding means decodes the quantization parameter of the subblock containing the significance coefficient first among the four subblocks in the decoding order, and sets the decoded quantization parameter as the quantization parameter of the other subblocks containing the significance coefficient among the four subblocks.

15. The image decoding apparatus according to any one of claims 9 to 14, characterized in that the four subblocks are subblocks obtained by quadruple-tree partitioning of square subblocks generated by dividing a rectangular subblock.

16. The image decoding apparatus according to claim 15, characterized in that, if the size information is greater than a quantization control threshold indicating a unit for encoding the quantization parameters, the decoding means shares the quantization parameters decoded in horizontally adjacent subblocks if the plurality of subblocks that share the quantization parameters to which the immediately decoded subblock belongs are horizontally elongated rectangles, and shares the quantization parameters decoded in vertically adjacent subblocks if the plurality of subblocks that share the quantization parameters to which the immediately decoded subblock belongs are vertically elongated rectangles.

17. An image coding method comprising: a first division step of dividing an image into a plurality of blocks; a second division step of dividing the blocks into a plurality of subblocks; a prediction step of calculating a prediction error based on a prediction image generated by performing a prediction process on the subblock units; a quantization step of quantizing the prediction error using quantization parameters to generate quantization coefficients; and an encoding step of encoding the quantization coefficients and the quantization parameters, wherein, in the prediction step, the subblock is divided into four subblocks, the quantization step determines whether or not to share the quantization parameters among the four subblocks based on a comparison between size information corresponding to the size of the four subblocks and a quantization control threshold indicating a unit for encoding the quantization parameters, and the encoding step switches the encoding process of the quantization parameters of the four subblocks based on the determination.

18. An image decoding method for decoding an image by decoding a bitstream generated by encoding each of several subblocks obtained by dividing an image into multiple blocks, the method comprising: a separation step of separating coded data from the bitstream; a decoding step of decoding the quantization parameters of the subblocks from the coded data; an inverse quantization step of generating a prediction error by inverse quantization of the quantization coefficients based on the quantization parameters; and an image playback step of reproducing an image based on the prediction error, wherein in the decoding step, when the target of decoding the quantization parameters is four subblocks, the decoding process for the quantization parameters of the four subblocks is switched based on a comparison between size information corresponding to the size of the four subblocks and a quantization control threshold indicating the unit in which the quantization parameters are encoded.

19. A program for causing a computer to function as one of the means of an image encoding apparatus according to any one of claims 1 to 8.

20. A program for causing a computer to function as one of the means of an image decoding apparatus according to any one of claims 9 to 16.

Images