Moving image encoding device and decoding device

Abstract

A conventional method using packed region information has had the problem that, when there is no information the width and height of an image to be reconstructed, the image cannot be correctly reconstructed.　A moving image decoding device according to one aspect of the present invention is characterized by comprising an image decoding unit that decodes encoded data obtained by encoding a packed image which is obtained by selecting a rectangular region of interest from an image, changing the resolution thereof, and repacking the region of interest into a smaller screen, a packed region information decoding unit that decodes packed region information which is information for performing reconstruction from the packed image to the original image, and a packed image reconstruction unit that reconstructs the packed image on the basis of the packed region information, wherein, when there is no information regarding the width and height of an image to be reconstructed in the packed region information, the width and height of a maximum display image of the encoded data are used.

Description

Moving picture encoding apparatus and decoding apparatus

[0001] Embodiments of the present invention relate to a moving picture encoding apparatus and a decoding apparatus.

[0002] In order to efficiently transmit or record moving pictures, a moving picture encoding apparatus that generates encoded data by encoding an image, and a moving picture decoding apparatus that generates a decoded image by decoding the encoded data are used.

[0003] Specific moving picture encoding schemes include, for example, the H.266 / VVC (Versatile Video Coding) scheme.

[0004] In such conventional image encoding schemes, an image is divided for encoding / decoding. First, a prediction image is generated based on a locally decoded image obtained by encoding an input image and decoding encoded data. Next, a prediction error (sometimes referred to as a "difference image" or "residual image") obtained by subtracting the prediction image from the input image (original image) is encoded / decoded.

[0005] In recent years, in moving picture encoding and decoding systems, not only encoding, transmission and decoding of video content for human viewing, but also research and development on moving picture encoding and decoding systems for machines for the purpose of image recognition have been conducted.

[0006] In non-patent literature, as a technology for moving picture encoding and decoding, a supplemental enhancement information SEI (Supplemental Enhancement Information) message for transmitting image properties, display methods, timing and other information together with encoded data is specified.

[0007] Here, a method is used in which a rectangular region of interest (ROI) is selected from the entire screen, its resolution is changed, and it is repacked into a smaller screen and encoded as a lower-resolution image. As a result, a high-quality decoding result is obtained only for the region of interest, reducing the overall bitrate and pixel count, and alleviating the complexity of the decoding process and post-processing operations such as image recognition. The Packed Region Information (PRI) SEI message, as defined in non-patent literature, is an auxiliary extended SEI that specifies a method for reconstructing the packed image region into an image in order to realize the above method.

[0008] J. Boyce, J. Chen, S. Deshpande, MM Hannuksela, S. McCarthy, GJ Sullivan, H. Tan, Y.-K. Wang, Additional SEI messages for VSEI version 4 (Draft 5), JVET-AK2006, Geneva, Feb.15 2025.

[0009] Methods disclosed in non-patent literature had the problem of not being able to correctly reconstruct an image if the width and height information of the image to be reconstructed was not available. Therefore, it was difficult to apply this method to the encoding and decoding of moving images for machines.

[0010] A motion image decoding device according to one aspect of the present invention includes: an image decoding unit that decodes encoded data obtained by selecting a rectangular area of ​​high interest from an image, changing the resolution, and repacking it into a smaller screen; a packed area information decoding unit that decodes packed area information, which is information for reconstructing the original image from the packed image; and a packed image reconstruction unit that reconstructs the packed image based on the packed area information, characterized in that, if the packed area information does not contain information on the width and height of the reconstructed image, the width and height of the largest display image in the encoded data are used.

[0011] Furthermore, a motion image encoding device according to one aspect of the present invention includes a stuffed image processing unit that selects a rectangular area of ​​high interest from an input image, changes the resolution, and creates a stuffed image by rearranging it to fit on a smaller screen; an image decoding unit that encodes the stuffed image and outputs encoded data; a stuffed area information encoding unit that encodes stuffed area information, which is information for reconstructing the original image from the stuffed image; and, if the stuffed area information does not contain information on the width and height of the reconstructed image, the device uses the width and height of the largest display image in the encoded data.

[0012] This configuration solves the challenge of realizing a video encoding and decoding system for machines.

[0013] This is a schematic diagram showing the configuration of the image transmission system according to this embodiment. This is a diagram showing an example of a block diagram of the packed image processing unit according to this embodiment. This is a diagram showing an example of packed image processing according to this embodiment. This is a diagram (1) showing the syntax of the Packed Region Information SEI message from the non-patent literature. This is a diagram (2) showing the syntax of the Packed Region Information SEI message from the non-patent literature. This is a diagram showing the relationship between the variable ChromaFormatIdc and the variables SubWidthC and SubHeightC. This is a diagram showing the syntax of the Packed Region Information SEI message in this embodiment.

[0014] (First Embodiment) Figure 1 is a conceptual diagram showing the configuration of the image transmission system for machines according to this embodiment.

[0015] The image transmission system 1 consists of a video encoding device 10, a transmission network 20, a video decoding device 30, an image display device 40, and an image recognition device 50.

[0016] The video encoding device 10 takes an input image signal T as input and outputs encoded data Te.

[0017] The transmission network 20 transmits encoded data Te from the video encoding device 10 to the video decoding device 30. The transmission network 20 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 20 is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, the transmission network 20 may be replaced by a storage medium that records encoded data Te, such as a DVD (Digital Versatile Disc: trademark) or a BD (Blu-ray Disc: registered trademark).

[0018] The video decoding device 30 takes encoded data Te as input, outputs a decoded image Td, and sends it to the image display device 40. The video decoding device 30 also sends the decoded image Td to the image recognition device 50.

[0019] The image display device 40 displays all or part of the decoded image Td output from the video decoding device 30. The image display device 40 includes a display device such as a liquid crystal display or an organic EL (electroluminescence) display. Examples of display forms include stationary, mobile, and HMD. Furthermore, if the video decoding device 30 has high processing power, it displays a high-quality image, and if it has lower processing power, it displays an image that does not require high processing power or display power.

[0020] The image recognition device 50 processes the decoded image Td and performs image recognition. Specifically, it identifies what objects are in the image, performs tracking of the identified objects when they are converted into moving images, and performs segmentation of different regions within the image.

[0021] The video encoding device 10 consists of a packed image processing unit 101, an image encoding unit 102, and a packed area information encoding unit 103.

[0022] The stuffed image processing unit 101 receives an input image signal T, outputs the stuffed image to the image encoding unit 102, and outputs the stuffed area information to the stuffed area information encoding unit 103. The image encoding unit 102 encodes the stuffed image created by the stuffed image processing unit 101 and creates encoded data Te. For example, video encoding and decoding schemes such as AVC, HEVC, and VVC may be applied, respectively.

[0023] The packed area information encoding unit 103 takes the input image signal T and the packed area information from the packed image processing unit 101 as input, encodes the packed area information as SEI, and creates encoded data Te.

[0024] The video decoding device 30 consists of an image decoding unit 301, a packed area information decoding unit 302, and a packed image reconstruction unit 303.

[0025] The image decoding unit 301 receives live encoded data Te transmitted via the transmission network 20 as input, performs decoding processing, and sends the decoded packed image and decoding information to the packed image reconstruction unit 303.

[0026] The packed area information decoding unit 302 decodes the encoded data Te based on its syntax and sends it to the packed image reconstruction unit 303.

[0027] The packed image reconstruction unit 303 reconstructs the decoded packed image based on the packed area information to create a decoded image Td.

[0028] Figure 2 is a conceptual diagram showing the configuration of the packed image processing unit 101 in this embodiment. The packed image processing unit 101 consists of an image recognition unit 1011 and a packed area image processing unit 1012.

[0029] The packed image processing unit 101 selects a rectangular region of interest (ROI) from the input image signal, changes the resolution, and repackages it into a smaller screen to generate a packed image with a lower resolution.

[0030] The image recognition unit 1011 recognizes and extracts a rectangular region of interest (ROI) from the input image signal. The stuffed region image processing unit 1012 resizes and rearranges the image region to create a stuffed image and sends it to the image encoding unit 102. It also sends stuffed region information, which is information for reconstructing the original image from the stuffed image, to the stuffed region encoding unit 103.

[0031] Figure 3 shows an example of the generation of a packed image by the packed image processing unit 101 of this embodiment. Figure 3(a) is an image of the input image signal. Region C is the entire image of the input image signal, and rectangular regions A and B are regions of interest. Figure 3(b) is an example of a packed image. The size of region C is reduced and rearranged with regions A and B to create a smaller packed image.

[0032] The packed image reconstruction unit 303 reconstructs regions A, B, and C from the decoded image, decoding information, and packed region information.

[0033] By encoding and decoding such packed images, high-quality decoding results are obtained only for the areas of interest, in this case regions A and B. Overall, the bitrate and pixel count are reduced, and the complexity of the decoding process and post-processing operations such as image recognition is reduced.

[0034] In the image recognition device 50, performing image recognition processing on the reconstructed decoded image Td ensures a high recognition rate and accuracy because the image quality of the area of ​​interest is good.

[0035] Alternatively, it is possible to input the decoded image of the packed image before reconstruction into the image recognition device 50 and perform image recognition processing. In this case, since the image processing involves a small number of pixels, the amount of computation can be reduced.

[0036] In this embodiment, the data is encoded and decoded as multiple SEI (Supplemental Enhancement Information) messages based on the syntax described later. Note that the encoding and decoding method is not limited to SEI messages; it may also be encoded and decoded as a syntax within a video encoding and decoding scheme.

[0037] Next, a Packed Region Information (PRI) SEI message described in non-patent literature will be described.

[0038] Note that the meaning of the Descriptor notation in the following syntax tables shall be interpreted as follows.

[0039] ・b(8): represents a byte value having any pattern of a bit string (8 bits).

[0040] ・f(n): represents a fixed-pattern bit string using n bits written in order from the left bit (from left to right).

[0041] ・se(v): represents a syntax element obtained by 0-th order Exp-Golomb coding of a signed integer.

[0042] ・st(v): represents a character string encoded in UTF-8 and null-terminated.

[0043] ・u(n): represents an unsigned integer using n bits. When n is "v" in a syntax table, the number of bits varies depending on the values of other syntax elements.

[0044] ・ue(v): represents a syntax element obtained by 0-th order Exp-Golomb coding of an unsigned integer, with the left bit first.

[0045] Figures 4 and 5 show the syntax of the packed area information SEI from the non-patent literature for encoding and decoding packed area information. The following variables are defined for the interpretation of the packed area information SEI message: The variable PicWidthInLumaSamples is set to equal the value of pps_pic_width_in_luma_samples-SubWidthC*(pps_conf_win_left_offset+pps_conf_win_right_offset). The variable PicHeightInLumaSamples is set to equal the value of pps_pic_height_in_luma_samples-SubHeightC*(pps_conf_win_top_offset+pps_conf_win_bottom_offset). The variable MaxWidth is set to equal the value of sps_pic_width_in_luma_samples-SubWidthC*(sps_conf_win_left_offset+sps_conf_win_right_offset). - The variable MaxHeight is set to equal the value of sps_pic_height_in_luma_samples-SubHeightC*(sps_conf_win_top_offset+sps_conf_win_bottom_offset). - The variable ChromaFormatIdc is set to the same value as sps_chroma_format_idc. sps_chroma_format_idc is an SPS syntax element that defines the color difference format by value. - Both the variables BitDepthY and BitDepthC are set to the same value as sps_bitdepth_minus8 plus 8.

[0046] Here, pps_pic_width_in_luma_samples is one of the Picture Component Set (PPS) syntax elements, indicating the number of pixels in the horizontal direction of the luminance picture to be decoded.

[0047] pps_conf_win_left_offset and pps_conf_win_right_offset are syntax elements of PPS, which respectively indicate the left offset value and the right offset value for display.

[0048] pps_pic_height_in_luma_samples is one of the syntax elements of the Picture Parameter Set (PPS), which indicates the number of vertical pixels of the decoded luma picture.

[0049] pps_conf_win_top_offset and pps_conf_win_bottom_offset are syntax elements of PPS, which respectively indicate the upper offset value and the lower offset value for display.

[0050] sps_pic_width_in_luma_samples is one of the syntax elements of the Sequence Parameter Set (SPS), which indicates the maximum number of horizontal pixels of the decoded luma picture.

[0051] sps_conf_win_left_offset and sps_conf_win_right_offset are syntax elements of SPS, which respectively indicate the left offset value and the right offset value for display.

[0052] sps_pic_height_in_luma_samples is one of the syntax elements of the Sequence Parameter Set (SPS), which indicates the maximum number of vertical pixels of the decoded luma picture.

[0053] sps_conf_win_top_offset and sps_conf_win_bottom_offset are syntax elements of SPS, which respectively indicate the upper offset value and the lower offset value for display.

[0054] sps_bitdepth_minus8 is a syntax element of SPS, and the value obtained by adding 8 to this syntax element indicates the bit length of luma and chroma pixels.

[0055] The value of the variable ChromaFormatIdc indicates the format: 0 for monochrome format, 1 for 4:2:0 format, 2 for 4:2:2 format, and 3 for 4:4:4 format. Figure 6 illustrates the variables SubWidthC and SubHeightC, which define the horizontal and vertical number of chrominance pixels in a chrominance format. When the value of ChromaFormatIdc is 0, both SubWidthC and SubHeightC are 1; when the value is 1, both SubWidthC and SubHeightC are 2; when the value is 2, SubWidthC is 2 and SubHeight is 1; and when the value is 3, both SubWidthC and SubHeightC are set to 1.

[0056] Furthermore, the following variables must be defined in order to use this SEI message.

[0057] The width and height of the packed image per luminance pixel are shown in PicWidthInLumaSamples and PicHeightInLumaSamples, respectively.

[0058] The maximum image width and maximum image height per luminance pixel are indicated by MaxPicWidth and MaxPicHeight, respectively.

[0059] The variable ChromaFormatIdc, which indicates the color difference format, has the following values: 0 for monochrome format, 1 for 4:2:0 format, 2 for 4:2:2 format, and 3 for 4;4;4 format.

[0060] - The bit length of the pixels in the luminance component (indicated here as BitDepthY), and, if ChromaFormatIdc is not 0, the bit length of the pixels in the two related chrominance components (indicated here as BitDepthC).

[0061] Next, we will explain the semantics of the syntax elements in Figures 4 and 5.

[0062] A value of pri_cancel_flag in the syntax element indicates that the SEI message cancels the persistence of previous packing area information SEI messages in the output order applied to the current hierarchy. A value of pri_cancel_flag indicates that packing area information continues. If the value of pri_cancel_flag is 0, the following syntax elements exist.

[0063] The syntax element pri_persistence_flag specifies the persistence of the packed area information SEI message for the current hierarchy. A value of pri_persistence_flag of 0 specifies that the packed area information applies only to the currently decoded image. A value of pri_persistence_flag of 1 indicates that the packed area information SEI message applies to the currently decoded image and persists to all subsequent images in the current hierarchy in output order until one or more of the following conditions are met.

[0064] - A new CLVS (Coded Video Sequence) will be started in the current hierarchy.

[0065] - The bitstream ends.

[0066] - The image of the current hierarchy within the AU (Access Unit) associated with the packed area information SEI message is output after the current image in the output order.

[0067] The value obtained by adding 1 to the syntax element pri_num_regions_minus1 specifies the number of regions to which information is notified.

[0068] A value of 1 for the syntax element pri_multilayer_flag indicates that the syntax element pri_region_layer_id[i] exists. A value of 0 for pri_multilayer_flag indicates that pri_region_layer_id[i] does not exist.

[0069] When the value of the syntax element pri_use_max_dimensions_flag is 1, it specifies that MaxPicWidth, MaxPicHeight, PicWidthInLumaSamples, and PicHeightInLumaSamples shall be used in the calculation of variables. When the value of pri_use_max_dimensions_flag is 0, it indicates that MaxPicWidth, MaxPicHeight, PicWidthInLumaSamples, and PicHeightInLumaSamples are not used in the calculation of region parameter variables.

[0070] The syntax element pri_log2_unit_size specifies the unit size used in the calculation of region parameter variables. The variable priUnitSize is set to the value of (1 << pri_log2_unit_size).

[0071] The value obtained by adding 1 to the syntax element pri_region_size_len_minus1 specifies the number of bits used for signaling pri_region_top_left_in_units_x[i], pri_region_top_left_in_units_y[i], pri_region_width_in_units_minus1[i], pri_region_height_in_units_minus1[i], pri_target_region_top_left_x[i], and pri_target_region_top_left_y[i].

[0072] When the value of the syntax element pri_region_id_present_flag is 1, it indicates that the syntax element pri_region_id[i] is present. When the value of pri_region_id_present_flag is 0, it indicates that the syntax element pri_region_id[i] is not present.

[0073] A value of 1 for the syntax element pri_target_pic_params_present_flag indicates that the syntax elements pri_target_region_top_left_x[i], pri_target_region_top_left_y[i], pri_target_pic_width_minus1, and pri_target_pic_height_minus1 exist. A value of 0 for pri_target_pic_params_present_flag indicates that pri_target_region_top_left_x[i], pri_target_region_top_left_y[i], pri_target_pic_width_minus1, and pri_target_pic_height_minus1 do not exist.

[0074] The values ​​obtained by adding 1 to the syntax element pri_target_pic_width_minus1 and the values ​​obtained by adding 1 to the syntax element pri_target_pic_height_minus1 represent the width and height of the image reconstructed from the pixels of the decoded packed image, respectively, in terms of the number of luminance pixels.

[0075] The value obtained by adding 1 to the syntax element pri_num_resampling_ratios_minus1 specifies the number of resampling ratios to be notified.

[0076] The value obtained by adding 1 to the syntax element pri_resampling_width_num_minus1[i] and the value obtained by adding 1 to the syntax element pri_resampling_width_denom_minus1[i] define the numerator and denominator of the width resampling for the i-th resampling ratio, respectively. Both pri_resampling_width_num_minus1[i] and pri_resampling_width_denom_minus1[i] must be in the range of 0 to 65535. If they do not exist, the values ​​of pri_resampling_ratio_width_num_minus1[0] and pri_resampling_ratio_width_denom_minus1[0] are assumed to be equal to 0.

[0077] If the syntax element pri_fixed_aspect_ratio_flag[i] is equal to 1, it indicates that the syntax elements pri_resampling_height_num_minus1[i] and pri_resampling_height_denom_minus1[i] do not exist. If the value of pri_fixed_aspect_ratio_flag[i] is 0, it specifies that the syntax elements pri_resampling_height_num_minus1[i] and pri_resampling_height_denom_minus1[i] exist.

[0078] The value obtained by adding 1 to the syntax element pri_resampling_height_num_minus1[i] and the value obtained by adding 1 to the syntax element pri_resampling_height_denom_minus1[i] represent the numerator and denominator of the i-th resampling ratio height resampling, respectively. Both pri_resampling_height_num_minus1[i] and pri_resampling_height_denom_minus1[i] must be in the range of 0 to 65535. If none exist, the values ​​of pri_resampling_height_num_minus1[i] and pri_resampling_height_denom_minus1[i] are assumed to be equal to pri_resampling_width_num_minus1[i] and pri_resampling_width_denom_minus1[i], respectively.

[0079] The syntax element pri_region_id[i] indicates the ID value of the i-th region. If it does not exist, the value of pri_region_id[i] is assumed to be equal to i.

[0080] pri_region_layer_id[i] defines the layer identifier of the image associated with the region information of pri_region_top_left_in_units_x[i], pri_region_top_left_in_units_y[i], pri_region_width_in_units_minus1[i], and pri_region_height_in_units_minus1[i]. If it does not exist, the value of pri_region_layer_id[i] is assumed to be equal to 0.

[0081] If pri_region_is_a_layer_flag[i] is equal to 1, then the width and height of the image in the hierarchy of pri_region_layer_id[i] are the same as the width and height of the region at index i, and pri_region_top_left_in_units_x[i], pri_region_top_left_in_units_y[i], pri_region_width_in_units_minus1[i], and pri_region_height_in_units_minus1[i] are not described. If they do not exist, the value of pri_region_is_a_layer_flag[i] is assumed to be equal to 0.

[0082] pri_region_top_left_in_units_x[i] and pri_region_top_left_in_units_y[i] define the horizontal and vertical positions of the top-left sample of the i-th region, respectively.

[0083] The length of a syntax element is pri_region_size_len_minus1+1.

[0084] The variables priRegionTopLeftX[i] and priRegionTopLeftY[i], which represent the horizontal and vertical positions of the top-left luminance pixel of the i-th region of the decoded image of the packed region with hierarchical identifier pri_region_layer_id[i], are derived as follows:

[0085] if( !pri_use_max_dimensions_flag ) { priRegionTopLeftX[i]=pri_region_top_left_in_units_x[i]*priUnitSize priRegionTopLeftY[i]=pri_region_top_left_in_units_y[i]*priUnitSize} else { priRegionTopLeftX[i]=(pri_region_top_left_in_units_x[i]*priUnitSize* PicWidthInLumaSamples+MaxPicWidth / 2) / MaxWidth priRegionTopLeftY[i]=(pri_region_top_left_in_units_y[i]*priUnitSize* PicHeightInLumaSamples+MaxPicHeight / 2) / MaxHeight} The values ​​obtained by adding 1 to the syntax element pri_region_width_in_units_minus1[i] and the values ​​obtained by adding 1 to the syntax element pri_region_height_in_units_minsu1[i] define the horizontal and vertical positions of the width and height of the i-th region, respectively, in units.

[0086] The length of a syntax element is pri_region_size_len_minus1+1.

[0087] The variables priRegionWidth[i] and priRegionHeight[i], which represent the width and height of the luminance pixels in the i-th region of the decoded image of the packed image, are derived as follows:

[0088] if( !pri_use_max_dimensions_flag ) { priRegionWidth[i]=(pri_region_width_in_units_minus1[i]+1)*priUnitSize priRegionHeight[i]=(pri_region_height_in_units_minus1[i]+1)*priUnitSize} else { priRegionWidth[i]=((pri_region_width_in_units_minus1[i]+1)*priUnitSize * PicWidthInLumaSamples+MaxPicWidth / 2) / MaxWidth priRegionHeight[i]=((pri_region_height_in_units_minus1[i]+1)*priUnitSize* PicHeightInLumaSamples+MaxPicHeight / 2) / MaxHeight} Variables SubWidthC and SubHeightC are specified in Figure 6 It is derived from ChromaFormatIdc.

[0089] Furthermore, as a requirement for bitstream compliance, priRegionWidth[i]%SubWidthC must be equal to 0, and priRegionHeight[i]%SubHeightC must be equal to 0.

[0090] The syntax element pri_resampling_ratio_idx[i] specifies the index of the resampling ratio used for the i-th region. The length of the syntax element is Ceil(Log2(pri_num_resampling_ratios_minus1+1)).

[0091] The variables priResampleWidthNum[i], priResampleWidthDenom[i], priResampleHeightNum[i], and priResampleHeightDenom[i] are derived as follows:

[0092] priResampleWidthNum[i]= pri_resampling_width_num_minus1[pri_rsampling_ratio_idx[i]]+1 priResampleWidthDenom[i]= pri_resampling_width_denom_minus1[pri_rsampling_ratio_idx[i]]+1 priResampleHeightNum[i]= pri_resampling_height_num_minus1[pri_rsampling_ratio_idx[i]]+1) priResampleHeightDenom[i]= pri_resampling_height_denom_minus1[pri_rsampling_ratio_idx[i]]+1 The syntax elements pri_target_region_top_left_x[i] and pri_target_region_top_left_y[i], if present, indicate the horizontal and vertical positions of the top-left pixel of the luminance pixels in the i-th region of the reconstructed target image, respectively. The variables priTargetRegionWidth and priTargetHeight, which represent the width and height of the luminance samples in the resampled region of the reconstructed target image, are derived as follows:

[0093] priTargetRegionWidth = Round(( priRegionWidth[i]*priResampleWidthNum[i])÷ (priResampleWidthDenom[i]*SubWidthC))*SubWidthC priTargetRegionHeight = Round((priRegionHeight[i]*priResampleHeightNum[i])÷ (priResampleHeightDenom[i]*SubHeightC))*SubHeightC When reconstructing the target image using a pixel array of brightness of size (pri_target_pic_width_minus1+1)×(pri_target_pic_height_minus1+1), all brightness pixel values ​​are initialized to value 1<<(BitDepthY-1), and if chrominance pixels exist, they are initialized to 1<<(BitDepthC-1).

[0094] For any sample position (x,y) and regions j and k, if all of the following conditions are met: • pri_region_id[j] > pri_region_id[k] - x is within (priRegionTopLeftX[j]..priRegionTopLeftX[j]+priRegionWidth[j]) - y is within (priRegionTopLeftY[j]..priRegionTopLeftY[j]+priRegionHeight[j]) - x is within (priRegionTopLeftX[k]..priRegionTopLeftX[k]+priRegionWidth[k]) - y is within (priRegionTopLeftY[k]..priRegionTopLeftY[k]+priRegionHeight[k]) - for pixels in the reconstructed image at a position (x, y) within the middle of (priRegionTopLeftX[k]..priRegionTopLeftX[k]+priRegionWidth[k]), if the region of pri_region_id[j] and the region of pri_region_id[k] overlap, the pixel value defined in the region of pri_region_id[j] is used.

[0095] One problem with the non-patent literature PRI SEI is that some of the syntax elements representing the width, height, and pixel positions of an image do not take color difference formats into account. In the case of a 4:2:0 format, since the color difference pixels are only half the size of the luminance pixels in both the horizontal and vertical directions, the pixel position, width, and height of the top-left corner of the region must be even in both the horizontal and vertical directions when expressed using the luminance pixel positions. Similarly, in the case of a 4:2:2 format, since the color difference pixels are only half the size of the luminance pixels in the horizontal direction, the pixel position and width of the top-left corner of the region must be even in the horizontal direction when expressed using the luminance pixel positions.

[0096] Therefore, in this embodiment, for the variables priRegionTopLeftX[i] and priRegionTopLeftY[i], which represent the horizontal and vertical positions of the top-left luminance pixel of the i-th region of the decoded image of the packed region whose hierarchical identifier is pri_region_layer_id[i], respectively, as a requirement for bitstream compliance, priRegionTopLeftX[i]%SubWidthC is set to be equal to 0 and priRegionTopLeftY[i]%SubHeightC is set to be equal to 0.

[0097] Imposing such constraints improves the problems associated with non-patent literature.

[0098] Alternatively, instead of limiting the values, change the derivation method of the variables priRegionTopLeftX[i], priRegionTopLeftY[i], priRegionWidth[i], and priRegionHeight[i] as follows: if( !pri_use_max_dimensions_flag ) { priRegionTopLeftX[i]=Round((pri_region_top_left_in_units_x[i]* priUnitSize)÷SubWidthC)*SubWidthC priRegionTopLeftY[i]=Round((pri_region_top_left_in_units_y[i]* priUnitSize)÷SubHeightC)*SubHeightC} else { priRegionTopLeftX[i]=Round((pri_region_top_left_in_units_x[i]* priUnitSize*PicWidthInLumaSamples)÷(MaxPicWidth*SubWidthC))*SubWidthC priRegionTopLeftY[i]=Round((pri_region_top_left_in_units_y[i]* priUnitSize*PicHeightInLumaSamples)÷ (MaxPicHeight*SubHeightC))*SubHeightC} if( !pri_use_max_dimensions_flag ) { priRegionWidth[i]=Round((pri_region_width_in_units_minus1[i]* priUnitSize)÷SubWidthC)*SubWidthC priRegionHeight[i]=Round((pri_region_height_in_units_minus1[i]* priUnitSize)÷SubHeightC)*SubHeightC} else { priRegionWidth[i]=Round((pri_region_width_in_units_minus1[i]*priUnitSize*PicWidthInLumaSamples)÷(MaxPicWidth*SubWidthC))*SubWidthC priRegionHeight[i]=(pri_region_height_in_units_minus1[i]*priUnitSize* PicHeightInLumaSamples)÷(MaxPicHeight*SubHeightC))*SubHeightC} With this method, priRegionTopLeftX[i]%SubWidthC will always be equal to 0, priRegionTopLeftY[i]%SubHeightC will always be equal to 0, priRegionWidth[i]%SubWidthC will always be equal to 0, priRegionHeight[i]%SubHeightC will always be equal to 0.

[0099] Alternatively, the calculation can be derived using integer division as follows.

[0100] if( !pri_use_max_dimensions_flag ) { priRegionTopLeftX[i]=((pri_region_top_left_in_units_x[i]* priUnitSize) / SubWidthC)*SubWidthC priRegionTopLeftY[i]=((pri_region_top_left_in_units_y[i]* priUnitSize) / SubHeightC)*SubHeightC } else { priRegionTopLeftX[i]=((pri_region_top_left_in_units_x[i]* priUnitSize*PicWidthInLumaSamples) / (MaxPicWidth*SubWidthC))*SubWidthC priRegionTopLeftY[i]=((pri_region_top_left_in_units_y[i]* priUnitSize*PicHeightInLumaSamples) / (MaxPicHeight*SubHeightC))*SubHeightC } if( !pri_use_max_dimensions_flag ) { priRegionWidth[i]=((pri_region_width_in_units_minus1[i]* priUnitSize) / SubWidthC)*SubWidthC priRegionHeight[i]=((pri_region_height_in_units_minus1[i]* priUnitSize) / SubHeightC)*SubHeightC } else { priRegionWidth[i]=((pri_region_width_in_units_minus1[i]* priUnitSize*PicWidthInLumaSamples) / (MaxPicWidth*SubWidthC))*SubWidthC priRegionHeight[i]=(pri_region_height_in_units_minus1[i]*priUnitSize*PicHeightInLumaSamples) / (MaxPicHeight*SubHeightC))*SubHeightC} With this method as well, priRegionTopLeftX[i]%SubWidthC will always be equal to 0, priRegionTopLeftY[i]%SubHeightC will be equal to 0, priRegionWidth[i]%SubWidthC will be equal to 0, and priRegionHeight[i]%SubHeightC will be equal to 0.

[0101] This will solve the problem.

[0102] Another problem with the PRI SEI described in non-patent literature is that the syntax elements pri_target_pic_width_minus1 and pri_target_pic_height_minus1, which indicate the width and height of the image reconstructed from the pixels of the decoded packed image in terms of the number of luminance pixels, do not take the color difference format into consideration.

[0103] Therefore, in this embodiment, as a requirement for bitstream compliance, the syntax elements and their values ​​must be such that (pri_target_pic_width_minus1+1)%SubWidthC is equal to 0 and (pri_target_pic_height_minus1+1)%SubHeightC is equal to 0.

[0104] This will solve the problem.

[0105] Another problem with the PRI SEI in the non-patent literature is that the syntax elements pri_target_region_top_left_x[i] and pri_target_region_top_left_y[i], which indicate the horizontal and vertical positions of the top-left pixel of the luminance pixel in the i-th region of the reconstructed target image, do not take the color difference format into consideration.

[0106] Therefore, in this embodiment, as a requirement for bitstream compliance, the values ​​of the syntax elements pri_target_region_top_left_x[i] and pri_target_region_top_left_y[i] must be such that pri_target_region_top_left_x[i]%SubWidthC is equal to 0 and pri_target_region_top_left_y[i]%SubHeightC is equal to 0.

[0107] This will solve the problem.

[0108] Alternatively, one could define priTargetTegionTopLeftX[i] and priTargetTegionTopLeftY[i] as variables representing the horizontal and vertical positions of the top-left pixel of the luminance pixel in the i-th region of the reconstructed target image, respectively, and derive the result as follows.

[0109] priTargetTegionTopLeftX[i]=(pri_target_region_top_left_x[i] / SubWidthC)*SubWidthC priTargetTegionTopLeftY[i]=(pri_target_region_top_left_y[i] / SubHeightC)*SubHeightC Even with this method, priTargetTegionTopLeftX[i]%SubWidthC will always be equal to 0, and priTargetTegionTopLeftY[i]%SubHeightC will always be equal to 0.

[0110] This will solve the problem.

[0111] Another problem with the PRI SEI in non-patent literature is that the initial value of the resampling ratio is not defined. This is because, in the syntax shown in Figure 4, the variable i in the loop for( i = 1; i <= pri_num_resampling_ratios_minus1; i++ ) { starts from 1, so the value when i is 0 is not defined.

[0112] In this embodiment, as shown in Figure 7, the variable i is used in a loop that starts from 0, as in for( i = 0; i <= pri_num_resampling_ratios_minus1; i++ ) { and the problem is solved.

[0113] This will solve the problem.

[0114] Another problem with the PRI SEI in non-patent literature is that when the value of the syntax element pri_target_pic_params_present_flag is 0, it indicates that pri_target_region_top_left_x[i], pri_target_region_top_left_y[i], pri_target_pic_width_minus1, and pri_target_pic_height_minus1 do not exist. However, in this case, the values ​​of pri_target_region_top_left_x[i], pri_target_region_top_left_y[i], pri_target_pic_width_minus1, and pri_target_pic_height_minus1 are not specified. Even when the value of pri_target_pic_params_present_flag is 0, pri_target_region_top_left_x[i], pri_target_region_top_left_y[i], pri_target_pic_width_minus1, and pri_target_pic_height_minus1 may be used, resulting in undefined behavior.

[0115] Therefore, in this embodiment, it is stipulated that when the value of pri_target_pic_params_present_flag is 0, pri_target_pic_width_minus1 is assumed to be equal to (MaxPicWidth-1), and pri_target_pic_height_minus1 is assumed to be equal to (MaxPicHeight-1).

[0116] At this time, the variable MaxPicWidth is set to equal the value of sps_pic_width_in_luma_samples-SubWidthC*(sps_conf_win_left_offset+sps_conf_win_right_offset), and the variable MaxPicHeight is set to equal the value of sps_pic_height_in_luma_samples-SubHeightC*(sps_conf_win_top_offset+sps_conf_win_bottom_offset). If there are multiple levels, the one with a level identifier of 0 is selected.

[0117] Furthermore, the syntax elements pri_target_region_top_left_x[i] and pri_target_region_top_left_y[i] are written regardless of the value of pri_target_pic_params_present_flag, as shown in the syntax in Figure 7.

[0118] This will solve the problem.

[0119] Another problem with the PRI SEI in non-patent literature was that if the syntax element pri_resampling_ratio_idx[i] did not exist, that is, if the value of the syntax element pri_num_resampling_ratios_minus1 was 0, the value of pri_resampling_ratio_idx[i] was not defined.

[0120] Therefore, in this embodiment, if the syntax element pri_resampling_ratio_idx[i] does not exist, its value is estimated to be 0.

[0121] This will solve the problem.

[0122] Another problem with the non-patent literature PRI SEI is that the definition of cases where multiple regions overlap in the reconstructed image is unclear. When defining the coordinates of the top-left corner, width, and height of the reconstructed rectangle, the coordinates of the horizontal region pri_region_id[i] are pri_target_region_top_left_x[i] + priTargetRegionWidth[i]-1, starting from pri_target_region_top_left_x[i], and the coordinates of the vertical region are pri_target_region_top_left_y[i] + priTargetRegionHeight[i]-1, starting from pri_target_region_top_left_y[j]. Therefore, in this embodiment, the case where the region of pri_region_id[j] and the region of pri_region_id[k] overlap is defined as follows.

[0123] For any sample position (x,y) and regions j and k, if all of the following conditions are met, the pixels of the reconstructed image at position (x,y) will use the pixel values ​​defined in region pri_region_id[j] if the region pri_region_id[j] and the region pri_region_id[k] overlap.

[0124] pri_region_id[j] > pri_region_id[k].

[0125] - x is within the range from pri_target_region_top_left_x[j] to pri_target_region_top_left_x[j] + priTargetRegionWidth[j] - 1.

[0126] - y is within the range from pri_target_region_top_left_y[j] to pri_target_region_top_left_y[j] + priTargetRegionHeight[j] - 1.

[0127] - x is within the range from pri_target_region_top_left_x[k] to pri_target_region_top_left_x[j] + priTargetRegionWidth[k] - 1.

[0128] - y is within the range from pri_target_region_top_left_y[k] to pri_target_region_top_left_y[j] + priTargetRegionHeight[k] - 1.

[0129] This will solve the problem.

[0130] Another problem with the PRI SEI in non-patent literature is that the reconstructed image may contain regions outside the frame.

[0131] In this embodiment, region i is defined as such that pri_target_region_top_left_x[i] + priTargetRegionWidth[i] - 1 must be less than or equal to pri_target_pic_width_minus1, and pri_target_region_top_left_y[i] + priTargetRegionHeight[i] - 1 must be less than or equal to pri_target_pic_height_minus1.

[0132] This will solve the problem.

[0133] Another problem with the PRI SEI in the non-patent literature is that while it processes the syntax element pri_multilayer_flag, it lacks processing for the syntax element pri_region_is_a_layer_flag[i].

[0134] Therefore, in this embodiment, a value of pri_multilayer_flag indicates that the syntax elements pri_region_layer_id[i] and pri_region_is_a_layer_flag[i] exist, and a value of pri_multilayer_flag indicates that pri_region_layer_id[i] and pri_region_is_a_layer_flag[i] do not exist.

[0135] This will solve the problem.

[0136] Another problem with the PRI SEI in non-patent literature is that while it specifies the existence of a hierarchy, the SEI itself assumes operation within a single hierarchy.

[0137] Therefore, in this embodiment, as shown in the syntax of Figure 7, the syntax elements pri_multilayer_flag, pri_region_layer_id[i], and pri_region_is_a_layer_flag[i] are deleted to set the hierarchical identifier to 0 and remove the multi-hierarchical coding correspondence.

[0138] This will solve the problem.

[0139] Regarding the issue of the non-patent literature not considering the PRI SEI color difference format, one method is to define the top-left coordinate, width, and height of the region using the pixel units of the color difference signal, rather than the pixel units of the luminance signal. Therefore, in this embodiment, instead of defining pre_log2_unit_size as in the non-patent literature, we use the syntax shown in Figure 7 and define the following semantics.

[0140] pri_region_top_left_in_units_x[i] and pri_region_top_left_in_units_y[i] define the horizontal and vertical positions of the top-left sample of the i-th region of the packed image.

[0141] The length of a syntax element is pri_region_size_len_minus1+1.

[0142] The value of pri_region_top_left_in_units_x[i] should be in the range of 0 to (PicWidthInLumaSamples / SubWidthC-1). If it does not exist, it is assumed to be 0.

[0143] The value of pri_region_top_left_in_units_y[i] should be in the range of 0 to (PicHeightInLumaSamples / SubHeightC-1). If it does not exist, it is assumed to be 0.

[0144] The variables priRegionTopLeftX[i] and priRegionTopLeftY[i], which represent the horizontal and vertical positions of the top-left luminance pixel of the i-th region of the decoded image of the packed region, are derived as follows:

[0145] if( !pri_use_max_dimensions_flag ) { priRegionTopLeftX[i]=pri_region_top_left_in_units_x[i]*SubWidthC priRegionTopLeftY[i]=pri_region_top_left_in_units_y[i]*SubHeightC} else { priRegionTopLeftX[i]=((pri_region_top_left_in_units_x[i]* PicWidthInLumaSamples+MaxPicWidth / 2) / MaxWidth)*SubWidthC priRegionTopLeftY[i]=((pri_region_top_left_in_units_y[i]* PicHeightInLumaSamples+MaxPicHeight / 2) / MaxHeight)*SubHeightC} The values ​​obtained by adding 1 to the syntax element pri_region_width_in_units_minus1[i] and the values ​​obtained by adding 1 to the syntax element pri_region_height_in_units_minsu1[i] define the width and height of the i-th region of the packed image.

[0146] The length of a syntax element is pri_region_size_len_minus1+1.

[0147] The value of pri_region_width_in_units_minus1[i] should be in the range of 0 to (PicWidthInLumaSamples / SubWidthC-pri_region_top_left_in_units_x[i]). If it does not exist, it is assumed to be PicWidthInLumaSamples / SubWidthC-1.

[0148] The value of pri_region_height_in_units_minus1[i] should be in the range of 0 to (PicHeightInLumaSamples / SubHeightC-pri_region_top_left_in_units_y[i]). If it does not exist, it is assumed to be PicHeightInLumaSamples / SubHeightC-1.

[0149] The variables priRegionWidth[i] and priRegionHeight[i], which represent the width and height of the luminance pixels in the i-th region of the decoded image of the packed image, are derived as follows:

[0150] if( !pri_use_max_dimensions_flag ) { priRegionWidth[i]=(pri_region_width_in_units_minus1[i]+1)*SubWidthC priRegionHeight[i]=(pri_region_height_in_units_minus1[i]+1)*SubHeightC} else { priRegionWidth[i]=(((pri_region_width_in_units_minus1[i]+1)* PicWidthInLumaSamples+MaxPicWidth / 2) / MaxWidth)*SubHeightC priRegionHeight[i]=(((pri_region_height_in_units_minus1[i]+1)* PicHeightInLumaSamples+MaxPicHeight / 2) / MaxHeight)*SubHeightC} The syntax elements pri_target_region_top_left_x[i] and pri_target_region_top_left_y[i] define the horizontal and vertical positions of the top-left pixel of the i-th region of the reconstructed target image.

[0151] The length of a syntax element is pri_region_size_len_minus1+1.

[0152] The value of pri_target_region_top_left_in_units_x[i] should be in the range of 0 to ((pri_target_pic_width_minus1+1) / SubWidthC-1). If it does not exist, it is assumed to be 0.

[0153] The value of pri_target_region_top_left_in_units_y[i] should be in the range of 0 to ((pri_target_pic_height_minus1+1) / SubHeightC-1). If it does not exist, it is assumed to be 0.

[0154] The variables priTargetRegionTopLeftX[i] and priTargetRegionTopLeftY[i], which represent the horizontal and vertical positions of the top-left brightness pixel of the i-th region of the reconstructed target image, are derived as follows:

[0155] The variables priTargetRegionWidth and priTargetHeight, which represent the width and height of the luminance samples in the resampled region of the reconstructed target image, are derived as follows: priTargetRegionTopLeftX[i]=pri_target_region_top_left_x[i]*SubWidthC priTargetRegionTopLeftY[i]=pri_target_region_top_left_y[i]*SubHeightC

[0156] priTargetRegionWidth[i] = Round(( priRegionWidth[i] * priResampleWidthNum[i]) ÷ (priResampleWidthDenom[i] * SubWidthC)) * SubWidthC priTargetRegionHeight[i] = Round((priRegionHeight[i] * priResampleHeightNum[i]) ÷ (priResampleHeightDenom[i] * SubHeightC)) * SubHeightC Note that, as a requirement for bitstream compliance, priTargetRegionWidth[i] must be in the range from 0 to (pri_target_pic_width_minus1 + 1 - priTargetRegionTopLeftX[i]). If it does not exist, it is estimated to be (pri_target_pic_width_minus1+1-priTargetRegionTopLeftX[i]), and priTargetRegionHeight[i] is set to a range from 0 to (pri_target_pic_height_minus1+1-priTargetRegionTopLeftY[i]). If it does not exist, it is estimated to be (pri_target_pic_height_minus1+1-priTargetRegionTopLeftY[i]).

[0157] For any sample position (x,y) and regions j and k, if all of the following conditions are met, the pixels of the reconstructed image at position (x,y) will use the pixel values ​​defined in region pri_region_id[j] if the region pri_region_id[j] and the region pri_region_id[k] overlap.

[0158] pri_region_id[j] > pri_region_id[k].

[0159] - x is within the range from priTargetRegionTopLeftX[j] to priTargetRegionTopLeftX[j]+priTargetRegionWidth[j]-1.

[0160] - y is within the range from priTargetRegionTopLeftY[j] to priTargetRegionTopLeftY[j]+priTargetRegionHeight[j]-1.

[0161] - x is within the range from priTargetRegionTopLeftX[k] to priTargetRegionTopLeftX[k]+priTargetRegionWidth[k]-1.

[0162] - y is within the range from priTargetRegionTopLeftY[k] to priTargetRegionTopLeftY[k]+priTargetRegionHeight[k]-1.

[0163] This will solve the problem.

[0164] In this embodiment, a packed area information encoding and decoding method using SEI is shown, but it can also be implemented by encoding and decoding the encoded data as an internal syntax of the video encoding and decoding method.

[0165] In summary, this embodiment demonstrates that by encoding and decoding with packed area information SEI messages, it becomes possible to control the image quality of a specific area, and that this is particularly useful for realizing a video transmission system using a video encoding and decoding method for machines.

[0166] Furthermore, some or all of the video encoding device 10 and video decoding device 30 in the above-described embodiment may be implemented using a computer. In that case, the program for implementing this control function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" refers to a computer system built into either the video encoding device 10 or the video decoding device 30, and includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. Moreover, "computer-readable recording medium" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such a case. Furthermore, the above-mentioned program may be for implementing some of the functions described above, and may also be able to implement the above-mentioned functions in combination with programs already recorded in the computer system.

[0167] Furthermore, some or all of the video encoding device 10 and video decoding device 30 in the above-described embodiment may be implemented as integrated circuits such as LSIs (Large Scale Integration). Each functional block of the video encoding device 10 and video decoding device 30 may be individually implemented as a processor, or some or all of them may be integrated into a single processor. Also, the method of implementing the integrated circuit is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if advances in semiconductor technology lead to the emergence of integrated circuit implementation technologies that can replace LSIs, integrated circuits using such technologies may be used. [Means for Solving the Problem]

[0168] A motion image decoding device according to one aspect of the present invention includes: an image decoding unit that decodes encoded data obtained by selecting a rectangular area of ​​high interest from an image, changing the resolution, and repacking it into a smaller screen; a packed area information decoding unit that decodes packed area information, which is information for reconstructing the original image from the packed image; and a packed image reconstruction unit that reconstructs the packed image based on the packed area information, characterized in that, if the packed area information does not contain information on the width and height of the reconstructed image, the width and height of the largest display image in the encoded data are used.

[0169] Furthermore, a motion image encoding device according to one aspect of the present invention includes a stuffed image processing unit that selects a rectangular area of ​​high interest from an input image, changes the resolution, and creates a stuffed image by rearranging it to fit on a smaller screen; an image decoding unit that encodes the stuffed image and outputs encoded data; a stuffed area information encoding unit that encodes stuffed area information, which is information for reconstructing the original image from the stuffed image; and, if the stuffed area information does not contain information on the width and height of the reconstructed image, the device uses the width and height of the largest display image in the encoded data.

[0170] Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of this invention.

[0171] The embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the claims. That is, embodiments obtained by combining technical means that have been appropriately modified within the scope of the claims are also included in the technical scope of the present invention.

[0172] Embodiments of the present invention can be suitably applied to a video decoding device that decodes encoded data in which an image signal has been encoded, and a video encoding device that generates encoded data in which image data has been encoded. Furthermore, they can be suitably applied to the data structure of encoded data generated by the video encoding device and referenced by the video decoding device.

[0173] 1 Image transmission system 10 Video encoding device 101 Packed image processing unit 1011 Image recognition unit 1012 Packed area image processing unit 102 Image encoding unit 103 Packed area information encoding unit 20 Transmission network 30 Video decoding device 301 Image decoding unit 302 Packed area information decoding unit 303 Packed image reconstruction unit 40 Image display device 50 Image recognition device

Claims

1. A motion image decoding device comprising: an image decoding unit that decodes encoded data obtained by selecting a rectangular area of ​​high interest from an image, changing the resolution, and repacking the packed image onto a smaller screen; a packed area information decoding unit that decodes packed area information, which is information for reconstructing the original image from the packed image; and a packed image reconstruction unit that reconstructs the packed image based on the packed area information, wherein if the packed area information does not contain information on the width and height of the reconstructed image, the device uses the width and height of the largest display image in the encoded data.

2. A video decoding device according to claim 1, characterized in that, if the packing area information does not contain information on the width and height of the reconstructed image, the device uses the width and height of the largest display image of the encoded data for which the hierarchical identifier is 0.

3. A motion image encoding device comprising: a packed image processing unit that selects a rectangular area of ​​high interest from an input image, changes the resolution, and creates a packed image by rearranging it to fit on a smaller screen; an image decoding unit that encodes the packed image and outputs encoded data; a packed area information encoding unit that encodes packed area information, which is information for reconstructing the original image from the packed image; and, if the packed area information does not contain information on the width and height of the reconstructed image, the device using the width and height of the largest display image in the encoded data.

Images