Video coding device, video decoding device, video coding method, and video decoding method

Abstract

In order to suppress an increase in the amount of processing required for pre-processing or post-processing in a video coding scheme that uses pre-processing and post-processing, this video coding device includes: a first acquisition means for acquiring first coordinates indicating the position of a partial region of a picture; a second acquisition means for acquiring second coordinates by processing the first coordinates to make the coordinate values even numbers; a third acquisition means for acquiring third coordinates on the basis of the partial region represented by the second coordinates; and a correction means for processing the third coordinates to make the coordinate values even numbers.

Description

Video Encoding Device, Video Decoding Device, Video Encoding Method, and Video Decoding Method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to efficiently transmit or record video, a video encoding device that generates an encoded representation (hereinafter referred to as a bitstream) by encoding an input video, and a video decoding device that decodes the bitstream to generate a decoded video are used.

[0003] As standards for video encoding formats, there are H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29. In addition, as recent video encoding technologies, the technology described in Non-Patent Document 1 can be cited.

[0004] In these video encoding formats, generally, irreversible processing is executed in the video encoding device. By using irreversible processing, the data amount of the generated bitstream is suppressed. However, the quality of the video (decoded video) decoded by the video decoding device deteriorates compared to the quality of the video (input video) input to the video encoding device.

[0005] Various formats have been developed for human viewing applications, including the above video encoding formats. Generally, the performance of these formats is evaluated from the perspective of how much the decoded video deteriorates with respect to the quality of the input video in terms of the quality of the video decoded from bitstreams of the same data amount generated according to the processing of each format.

[0006] In recent years, not only for human viewing, but the scenarios where video is used for machine processing are increasing. For example, as a process executed by a machine, there is an object detection process for detecting a person or the like included in the video.

[0007] Development of video encoding methods intended for machine processing is underway. For example, ISO / IEC / SC29 is working on standardizing a video encoding method for machines called Video for Coding Machines (VCM).

[0008] The performance of a video encoding scheme for machines is generally evaluated by the processing performance performed by the machine when using video decoded from a bitstream of a similar amount of data. Therefore, even if the quality of the decoded video is significantly degraded compared to the quality of the input video, the encoding scheme is considered to have good performance if the degree of degradation when using the decoded video is small compared to when using the input video. In other words, the difference in quality between the input video and the decoded video is not considered a problem. For this reason, video encoding devices that perform video encoding for machines may apply pre-processing that has not been used in schemes intended for human viewing, in order to improve the performance of the encoding scheme.

[0009] One preprocessing step to improve the performance of an encoding scheme is to replace the pixel values ​​of areas outside the region of interest in the input video, i.e., non-interest regions, with arbitrary pixel values. The region of interest is the area in the video that should be emphasized (important area). Hereafter, the region of interest may be referred to as ROI (Region of Interest). Also, objects such as people that should be emphasized in the region of interest will simply be referred to as objects. Non-Patent Document 2 discloses an example of preprocessing for machine-oriented video encoding, which includes the process of replacing the pixel values ​​of non-interest regions with arbitrary pixel values. Non-Patent Document 2 also shows an example of postprocessing when encoding and decoding video for machines.

[0010] Hereinafter, the process of reducing the amount of encoded data based on the location and size of the region of interest is referred to as ROI processing or RTG processing. In ROI processing (RTG processing), for example, pixel values ​​in the region of non-interest are replaced with arbitrary pixel values ​​(for example, pixel values ​​corresponding to gray). Furthermore, the method disclosed in Non-Patent Document 3 utilizes a process that divides the picture to be processed into one or more rectangular regions based on the location of the region of interest and deletes the rectangular region corresponding to the region of non-interest. When this process is applied, the region of interest in the picture moves to the upper left.

[0011] Furthermore, in a method that allows referencing the H. 274 / VSEI standard, the coordinates of the region of interest detected within the picture can be transmitted as auxiliary information called annotated_regions, as defined in the H. 274 / VSEI standard.

[0012] "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)", JVET-AD2030, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Region-of-Interest-Based Video Coding for Machines", IEEE International Conference on Multimedia and Expo Workshops, 2024

[0013] When transmitting video signals, video encoders generally encode signals in the YUV color space. YUV color space signals are not encoded in a format where the number of luminance component samples and chrominance component samples are the same (YUV4:4:4 format). Instead, to improve compression efficiency, they are encoded in a format where the number of chrominance component samples is reduced. One such format is the YUV4:2:0 format. In this reduced format, the number of chrominance component samples is less than the number of luminance component samples.

[0014] On the other hand, the processing units within video encoders and decoders are sometimes designed on the assumption that the number of samples for the luminance component and the chrominance component of the signal input to the processing unit are the same. When YUV4:2:0 format video is input to such a processing unit, pre-processing is performed to make the number of samples for the chrominance component the same as the number of samples for the luminance component, i.e., format conversion from YUV4:2:0 to YUV4:4:4 format is performed. In addition, format conversion from YUV4:4:4 to YUV4:2:0 format is applied as post-processing. Hereafter, these processes will be referred to as format conversion. Format conversion not only increases the processing load for pre-processing and post-processing, but also increases the number of samples to be processed in the chrominance component, thus increasing the processing load in the processing unit.

[0015] The problem of increased processing load can be solved by modifying the processing content of each processing unit so that it can be executed without applying format conversion to the input signal. However, if the correspondence between the position of the luminance component sample and the position of the corresponding chrominance component sample is not considered, the processing applied to the luminance component sample and the processing applied to the chrominance component sample may differ. For example, in a process that applies scaling to a part of an image, the processing load may increase if the scaling is done by different magnifications. Also, a phase shift may occur between the luminance component sample and the chrominance component sample, which may degrade the quality of the processed signal.

[0016] The present invention aims to provide a video encoding device, a video decoding device, a video encoding method, a video decoding method, a video encoding program, and a video decoding program that can suppress the increase in processing load in a video encoding scheme for video in a format in which samples of color difference components have been downsampled.

[0017] The video decoding method based on this disclosure obtains a first coordinate indicating the position of a subregion in a picture, obtains a second coordinate by performing a process to make the coordinate value of the first coordinate an even number, obtains a third coordinate based on the subregion represented by the second coordinate, and performs a process to make the coordinate value of the third coordinate an even number.

[0018] The video encoding method based on this disclosure obtains a first coordinate indicating the position of a subregion in a picture, obtains a second coordinate by performing a process to make the coordinate value of the first coordinate an even number, obtains a third coordinate based on the subregion represented by the second coordinate, and performs a process to make the coordinate value of the third coordinate an even number.

[0019] The video decoding device according to this disclosure includes: a first acquisition means for acquiring a first coordinate indicating the position of a partial region in a picture; a second acquisition means for acquiring a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; a third acquisition means for acquiring a third coordinate based on the partial region represented by the second coordinate; and a correction means for performing a process to make the coordinate value of the third coordinate an even number.

[0020] The video encoding apparatus according to this disclosure includes: a first acquisition means for acquiring a first coordinate indicating the position of a partial region in a picture; a second acquisition means for acquiring a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; a third acquisition means for acquiring a third coordinate based on the partial region represented by the second coordinate; and a correction means for performing a process to make the coordinate value of the third coordinate an even number.

[0021] According to the present invention, in a video encoding scheme that processes video in a format in which samples of color difference components have been downsampled, the increase in processing load is suppressed.

[0022] This is a block diagram showing embodiments of a video encoder and a video decoder. This is an explanatory diagram showing an example of the configuration and operation of the RTG processing unit. This is a block diagram showing an example of the configuration and operation of the RTG processing unit in a video encoder. This is an explanatory diagram showing an example of syntax. This is an explanatory diagram showing the correspondence between index and magnification. This is a block diagram showing an example of the configuration of the RTG processing unit in a video decoder. This is an explanatory diagram showing an example of processing by the RTG processing unit. This is an explanatory diagram schematically showing RTG processing. This is a block diagram showing an example of the configuration of the RTG processing unit in a video encoder. This is a block diagram showing an example of the configuration of the RTG processing unit in a video decoder. This is an explanatory diagram for explaining processing by the coordinate correction unit. This is a flowchart showing an example of the operation of a video encoder. This is a flowchart showing an example of the operation of a video decoder. This is an explanatory diagram for explaining processing by the coordinate correction unit. This is a flowchart showing an example of the operation of a video encoder. This is a flowchart showing an example of the operation of a video decoder. This is an explanatory diagram for explaining the effect of a modified example. This is a block diagram showing an example of the configuration of an information processing system. This is a block diagram showing the main parts of a video encoding device. This is a block diagram showing the main parts of a video decoding device.

[0023] In addition to RTG processing, the following technologies can be cited as elemental technologies for pre-processing and post-processing: • Time scalability • Spatial scalability • Bit depth shift (bit depth truncation)

[0024] Temporal scalability is a technique in which the video encoder reduces the frame rate in the temporal direction, and the video decoder upsamples the frame rate. In other words, it is a technique in which the video encoder performs frame decimation, and the video decoder performs frame interpolation. Temporal frame decimation and interpolation are sometimes referred to as TR (Temporal Re-sampling).

[0025] Spatial scalability is a technique in which the video encoder reduces the spatial resolution of the video, and the video decoder increases the resolution (returning it to the original resolution). In other words, it is a technique in which the video encoder shrinks the picture, and the video decoder enlarges the picture. Hereafter, the reduction and enlargement of the picture may be expressed as SR (Spatial Re-sampling) or spatial resampling.

[0026] Specifically, in SR, a reduction process is applied to the input video on a picture-by-picture basis so that the size (width and height) of the picture to be processed is reduced. In other words, in SR, the video encoder performs spatial resampling on the picture to be processed. By reducing the number of pixels to be encoded, it becomes easier to reduce the amount of data in the bitstream. During decoding, as a corresponding post-processing step, an enlargement process to the original size is applied to the decoded video. In other words, in SR, the video decoder performs spatial reconstruction on the picture to be processed.

[0027] In general, the sensitivity of object detection to bit depth is low when it comes to video encoding for machines. Therefore, bit depth shifting is used to reduce the number of bits used to represent the image.

[0028] Figure 1 is a block diagram showing an example of a video encoder 100 and a video decoder 200 suitable for a video encoding scheme for machines. Specifically, it is a block diagram showing an example of the configuration of a video encoder 100 that performs pre-processing and a video decoder 200 that performs post-processing.

[0029] The video encoder 100 includes a time resampling unit 101, an RTG (retargeting) processing unit 102, a spatial resampling unit 103, a bit depth shift unit 104, an internal encoder 105, and a multiplexer 106.

[0030] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectional flow. This is also true for other block diagrams.

[0031] The time resampling unit 101 performs time resampling on the input video (input picture). The time resampling performed by the time resampling unit 101 is, for example, frame decimation. The RTG processing unit 102 performs RTG processing on the frame decimated input video. The spatial resampling unit 103 applies spatial scalability to the output of the RTG processing unit 102. The spatial scalability performed by the spatial resampling unit 103 is, for example, picture reduction. The bit depth shifting unit 104 applies bit depth truncation to the output of the spatial resampling unit 103. The bit depth truncation performed by the bit depth shifting unit 104 is, for example, a 1-bit right shift of the pixel value.

[0032] The internal encoder 105 encodes the picture supplied from the bit depth shift unit 104 using a predetermined video encoding scheme. As the predetermined video encoding scheme, for example, an encoding scheme based on H.266 / VVC can be used. As an example, the predetermined video encoding scheme can be VVC with End2End added.

[0033] The multiplexer 106 outputs a bitstream that is a multiplexed bitstream of the video bitstream supplied from the internal encoder 105 and the bitstream of side information as auxiliary information.

[0034] The video decoder 200 includes a demultiplexer 201, an internal decoder 202, a spatial resampling unit 203, an RTG processing unit 204, a temporal resampling unit 205, and a bit depth shifting unit 206.

[0035] The demultiplexer 201 demultiplexes the bitstream input to the video decoder 200 to obtain the video bitstream and the side information bitstream. The demultiplexer 201 supplies the video bitstream to the internal decoder 202. The demultiplexer 201 supplies the side information bitstream to the RTG processing unit 204, etc.

[0036] The internal decoder 202 decodes the video bitstream and supplies the decoded picture (decoded video frame) to the spatial resampling unit 203.

[0037] The spatial resampling unit 203 applies spatial scalability to the decoded picture. The spatial scalability performed by the spatial resampling unit 203 is, for example, picture scaling. The RTG processing unit 204 performs RTG processing on the output of the spatial resampling unit 203. The temporal resampling unit 205 performs temporal resampling on the output of the RTG processing unit 204. The temporal resampling performed by the temporal resampling unit 205 is, for example, frame interpolation. The bit depth shift unit 206 reconstructs the image from the output of the temporal resampling unit 205. The bit depth shift unit 206 performs, for example, a 1-bit left shift of the pixel value.

[0038] Figure 2 is an explanatory diagram showing an example of the operation of the RTG processing units 102 and 204. It is assumed that the RTG processing units 102 and 204 perform RTG processing on a video signal in YUV4:4:4 format. It is also assumed that a picture in YUV4:2:0 format is input and output between each processing block in the video encoder 100 and the video decoder 200.

[0039] In Figure 2, W represents the width of the luminance component of the picture before RTG processing. H represents the height of the luminance component of the picture before RTG processing. W' represents the width of the luminance component of the picture after RTG processing. H' represents the height of the luminance component of the picture after RTG processing.

[0040] In that case, the RTG processing units 102 and 204 first upsample the color difference components (U and V components) of the input picture to convert it from YUV4:2:0 format to YUV4:4:4 format. Then, the RTG processing units 102 and 204 perform RTG processing. Finally, the RTG processing units 102 and 204 downsample the color difference components to convert the YUV4:4:4 format picture back to a YUV4:2:0 format picture.

[0041] When such processing is performed, the number of pixels processed for color difference components in RTG processing becomes four times greater than when RTG processing is performed using the YUV4:2:0 format. As a result, the processing load of RTG processing increases.

[0042] Next, an example of the RTG process will be described. FIG. 3 is a block diagram showing a configuration example and an operation example of the RTG processing unit in the video encoder.

[0043] FIG. 3 is a block diagram showing a configuration example of the RTG processing unit 102 in the video encoder 100. In the configuration example shown in FIG. 3, the RTG processing unit 102 includes an ROI information acquisition unit 1021, a partial region definition unit 1022, a parameter determination unit 1023, and a conversion unit 1024.

[0044] The ROI information acquisition unit 1021 acquires the position information of the ROI based on the information that can identify the ROI included in the control information input to the video encoder 100 from the outside. As an example, the coordinate information includes the x (horizontal direction, i.e., lateral direction) coordinate and the y (vertical direction, i.e., longitudinal direction) coordinate of the upper left of the region, and the size of the region (the size in the x direction and the size in the y direction). Note that the video encoder 100 may include an object detection unit that detects an object from a picture, and the object detection unit may set the region including the object as the ROI. As a method for detecting an object, for example, there is a method of using a neural network trained to detect a specific object. In this case, the neural network takes, for example, the pixel values of the input video picture as input and outputs the coordinates of the region determined to include the object. The output coordinates are used as the coordinates of the region of interest. The region of interest of each picture may be calculated only from the processing result of the corresponding picture, or may be calculated based on the processing results of a plurality of pictures. When calculated based on the processing results of a plurality of pictures, the ROI information acquisition unit 1021 buffers the pictures until a predetermined number of pictures are input to the RTG processing unit 204.

[0045] The partial region definition unit 1022 sets partial regions, for example, as follows.

[0046] The sub-region definition unit 1022 determines the division points in the picture based on the position information of the ROIs. Specifically, the sub-region definition unit 1022 defines the division points so that each ROI is divided at the boundary portion of the ROI's region. For example, the sub-region definition unit 1022 sets the coordinate values ​​of the horizontal division points to values ​​selected without overlap from the x-coordinates of the top-left and bottom-right points of each ROI. Similarly, the vertical division points are selected without overlap from the y-coordinates of the top-left and bottom-right points of each ROI. The sub-region definition unit 1022 then sets the arrays of the division point values ​​in each direction in ascending order as one-dimensional arrays Ph and Pv, respectively. Example (A) illustrates that six division points are defined in the horizontal direction and five division points are defined in the vertical direction. Note that the starting point (point with coordinate value 0) and ending point (point with coordinate value 400 or 300) are also included as division points. In this case, Ph = {0, 100, 220, 300, 360, 400} and Pv = {0, 50, 150, 250, 300}. Based on these division points, the regions in which the picture to be processed is divided into a grid are defined as subregions before the forward transformation. In this embodiment, the forward transformation is an affine transformation (enlargement or reduction, and translation).

[0047] The sub-region definition unit 1022 determines a reduction ratio (magnification) for each ROI and, based on the ROI magnification, determines the horizontal and vertical magnifications of the region between each division point, i.e., the horizontal and vertical magnifications of the divided region. Hereinafter, the horizontal and vertical magnifications of the region between division points may simply be referred to as the magnifications between division points. As a method for determining the ROI magnification, for example, the magnification of an ROI with a size greater than or equal to a predetermined value is set to greater than 0 and less than 1.0 (0 to 1.0), and the magnification of an ROI with a size less than a predetermined value is set to 1.0. Subsequently, the vertical magnification between division points is set to the maximum magnification among the magnifications of the ROIs located between each division point. If no ROI exists, 0 is set. Similarly, the vertical magnification between division points is determined from the magnifications of the ROIs located between each division point. The horizontal and vertical magnifications between division points are expressed as one-dimensional arrays Sh and Sv, respectively. In example (A), Sh = {0.0, 0.8, 0.0, 1.0, 0.0} and Sv = {0.0, 0.8, 1.0, 0.0}. Example (A) illustrates that the horizontal scaling factor of one ROI (the ROI containing the larger object) is set to 0.8 or 1.0, and the vertical scaling factor is set to 0.8. It also illustrates that the horizontal scaling factor of the other ROI (the ROI containing the smaller object) is set to 1.0, and the vertical scaling factor is set to 1.0.

[0048] The partial area definition unit 1022 calculates the split points after the forward transformation from the split points before the forward transformation and the magnification factors between those split points. Specifically, when expressing the split points in the horizontal direction after the forward transformation as a one-dimensional array Ph', Ph'[0]=0, Ph'[m]=Ph'[m-1]+(Ph[m]-Ph[m-1])*Sh[m-1] is calculated. Here, assuming the size of Ph is M and m is an integer greater than or equal to 1 and less than M. Similarly, for the one-dimensional array Pv' representing the split points in the vertical direction after the forward transformation, assuming the size of Pv is N and n is an integer greater than or equal to 1 and less than N, Pv'[0]=0, Pv'[n]=Pv'[n-1]+(Pv[n]-Pv[n-1])*Sv[n-1] is calculated. The size after the forward transformation is represented by the last elements of Ph' and Pv', that is, Ph'[M-1] and Pv'[N-1]. In the case of Example (A), Ph'={0, 0, 96, 96, 156, 156}, Pv'={0, 0, 80, 180, 180}, and the width and height after the forward transformation are 156 and 180 respectively. As described above, since the number of split points before and after the forward transformation is the same, the split points before and after the forward transformation can be associated with each other by elements with the same index. As described above, the size of the picture after the forward transformation is determined by the split points before the forward transformation and the magnification factors between those split points, that is, the position and magnification factor of the ROI. In other words, if the position or magnification factor of the ROI is different, the size of the picture after the forward transformation may change. As a method for aligning the sizes of the pictures in the sequence, for example, there is a method of updating the magnification factor between the split points so as to obtain a desired size.

[0049] The subregion definition unit 1022 includes the reduction ratio (magnification) of the ROI in the side information, along with the coordinate information of each ROI before affine transformation, i.e., the original coordinate information (corresponding to the division points shown in example (A)). The subregion definition unit 1022 also includes information that identifies the size of the picture before and after transformation in the side information. In the example shown in Figure 3, the magnification (0.8, 0.8) (the former is the horizontal magnification, the latter is the vertical magnification) is included in the side information along with the coordinate values ​​(100, 50, 220, 150) (the first two numbers are the x and y coordinates of the upper left of the region, and the latter two numbers are the x and y coordinates of the lower right of the region). In addition, the magnification (0.8, 1.0) is included in the side information along with the coordinate values ​​(100, 150, 220, 250). Furthermore, along with the coordinate values ​​(300, 50, 360, 250), the magnification (1.0, 1.0) is included in the side information.

[0050] As mentioned above, in this embodiment, in addition to the size of the picture before and after the forward transformation, i.e., before and after the RTG processing, the x and y coordinates of the top left corner and the size of the region are included in the side information. For example, the magnification is included in the side information as syntax shown in Figure 4. As shown in Figure 5, the correspondence between the index and the magnification is defined. The syntax elements shown in Figure 4 are as follows.

[0051] retargeting_flag: A flag indicating whether or not to perform RTG processing. retargeting_image_size_len: The number of bits for retargeting_image_width and retargeting_image_size_height. retargeting_image_size_width: The width of the picture after RTG processing in the video encoder. retargeting_image_size_height: The height of the picture after RTG processing in the video encoder. retargeting_image_size_difference_flag: A flag indicating whether or not the width or height of the picture differs before and after RTG processing. retargeting_to_output_difference_len: The number of bits for retargeting_to_output_difference_width and retargeting_to_output_difference_height. retargeting_to_output_difference_width: The difference in width of the picture before and after RTG processing in the video encoder. retargeting_to_output_difference_height: The difference in height of the picture before and after RTG processing in the video encoder. retargeting_roi_flag: A flag indicating whether ROI information is displayed. If 0, the entire picture is treated as the ROI, and its magnification is set to 1.0.

[0052] roi_size_len: Number of bits for roi_pos_x[i], roi_pos_y[i], roi_size_x[i], and roi_size_y[i]. num_rois_len: Number of bits for num_rois. num_rois: Number of ROIs. roi_scaling_factor_flag: Flag indicating whether scale_factor_idx[i] is explicitly specified. If 0, scale_factor_idx[i] = scale_factor_idx[i-1]. scale_factor_idx[i]: Index indicating the scaling factor of the i-th ROI. For example, it is specified as shown in the scale_factors_idx[i] table in Figure 5. roi_pos_x[i]: x-coordinate of the top left of the i-th ROI. roi_pos_y[i]: y-coordinate of the top left of the i-th ROI. roi_size_x[i]: Width of the i-th ROI. roi_size_y[i]: The height of the i-th ROI.

[0053] Note that u(n) is an n-bit unsigned integer. When n in the syntax table is "v", the number of bits changes depending on the values ​​of other syntax elements. These syntax elements may be set to a value for each picture and included in the side information, or a common value may be set for multiple pictures and included in the side information. Also, for example, the unit for setting each syntax element may be changed so that elements of syntax set before retargeting_roi_flag are included in the side information on a sequence basis, and elements set after roi_flag are included on a picture basis.

[0054] In the set of subregions through which encoded data will be transmitted, subregions other than subregion C (see Figure 3) are regions that encompass all or part of one or more ROIs. Hereinafter, the upper right subregion C, which does not contain an object, may be referred to as a specific region. For example, the subregion definition unit 1022 replaces the pixel values ​​of the specific region with fixed values ​​(for example, pixel values ​​corresponding to gray). In other words, with respect to a subregion that does not contain an object (corresponding to a type of non-ROI), the video encoder 100 does not delete the subregion that does not contain an object if a subregion containing an object exists in the same row or column. The specific region is then treated in the same way as an ROI.

[0055] Furthermore, the partial region definition unit 1022 sets the magnification of partial regions that do not contain objects (non-ROI) to 0.

[0056] Note that a subregion with a reduction ratio (magnification) of 1.0 means that the subregion will not be reduced. A subregion with a magnification of 0 means that the subregion will be deleted.

[0057] The parameter determination unit 1023 determines the parameters for the affine transformation for each subregion. That is, the parameter determination unit 1023 moves the subregion containing the ROI to the upper left of the picture and determines the parameters so that the subregion is reduced by the determined reduction ratio. The reduction ratio (magnification) of the subregion containing the ROI is transmitted to the video decoder 200 as side information about the parameters of the affine transformation, for example.

[0058] Then, the transformation unit 1024 performs the affine transformation operation for each sub-region based on the parameters of the affine transformation, as follows.

[0059] The transformation unit 1024 moves (translates) the subregion with a magnification of 1.0 by affine transformation. In the example shown in Figure 3, the other ROI (the ROI containing the smaller object) is translated.

[0060] The transformation unit 1024 shrinks and moves (translates) the subregions where the magnification is not 0 or 1.0 by affine transformation. In the example shown in Figure 3, one ROI (the ROI containing the larger object) is shrunk and translated.

[0061] In this way, a picture with ROI processing applied, as shown in example (B) in Figure 3, is created.

[0062] In this embodiment, the video encoder 100 does not transmit encoded data for the picture shown in example (A), but rather transmits encoded data for the set of subregions shown in example (B) (four subregions in example (B)). That is, the video encoder 100 transmits encoded data for subregions that include ROI, but does not transmit encoded data for subregions that do not include ROI.

[0063] Furthermore, the RTG processing unit 102 in the video encoder 100 also includes in the side information the size of the set of subregions after performing ROI processing including affine transformation (see example (B) in Figure 3), that is, the size of the set of subregions to which the encoded data is transmitted (since the coordinate value of the top left of the set is 0, it can be represented by the coordinate value of the bottom right), and information representing the relationship (difference) between the coordinates of the set and the original coordinates. The video encoder 100 may also include the number of ROIs (two in example (B)) in the side information.

[0064] Figure 6 is a block diagram showing an example of the configuration and operation of the RTG processing unit 204 in the video decoder 200. In the configuration example shown in Figure 6, the RTG processing unit 204 includes an ROI information acquisition unit 2041, a partial region restoration unit 2042, a parameter acquisition unit 2043, and a conversion unit 2044.

[0065] The ROI information acquisition unit 2041 acquires the ROI position information and ROI reduction ratio (magnification) from the side information obtained by demultiplexing, as well as the size of the picture before and after RTG processing in the video decoder 200.

[0066] The partial region reconstruction unit 2042 obtains division points from the size of the picture after RTG processing and the ROI position information. Referring to the example shown in Figure 6, the partial region reconstruction unit 2042 reconstructs the division points shown in example (A).

[0067] The parameter acquisition unit 2043 determines the parameters for the affine transformation from the side information. The parameter acquisition unit 2043 moves the subregion including the ROI to the original coordinate position and determines the parameters so that the subregion is enlarged by the reciprocal of the determined magnification. If the magnification is 0, the parameter acquisition unit 2043 sets the enlargement ratio to 0.

[0068] The conversion unit 2044 includes a pixel copy unit, a padding unit, and a region expansion unit. The pixel copy unit sets the pixel value of the decoded picture (the pixel value of the said region) as the pixel value of a subregion with a magnification of 1.0. Then, based on the ROI position information, the pixel copy unit uses an affine transformation to return the ROI to its original position (the original coordinate position before the affine transformation was applied in the video encoder 100). The pixel value of the subregion is the pixel value of each pixel within the subregion. In other words, setting a pixel value in a subregion means filling the subregion with pixels that have that pixel value, or in other words, setting pixels that have that pixel value in the subregion. Translation can also be achieved, for example, by copying data (pixels) from one region in memory to another region.

[0069] The padding unit determines the pixel values ​​to be set in a sub-region where the magnification in any direction is 0, and sets the determined pixel values ​​in that sub-region. In other words, the conversion unit 2044 pads the sub-regions where encoded data was not transmitted from the video encoder 100 with the determined pixel values. The pixel values ​​to be set in a sub-region with a magnification of 0 are determined, for example, as follows.

[0070] The padding unit calculates statistical values ​​of pixels adjacent to a subregion where encoded data was not transmitted (for example, the values ​​of pixels surrounding a subregion with a width of 1 pixel). The following example uses the case where the average value is used as the statistic. When the magnification of the i-th subregion horizontally and the j-th subregion vertically is 0, the padding unit treats pixels in the pre-transformation picture where the x-coordinate is between Ph'[i]-1 and P'h[i+1], and the y-coordinate is between Pv'[j]-1 and Pv'[j+1], as adjacent pixels before transformation. The padding unit calculates the average value of adjacent pixels in all subregions where the magnification is 0. Then, the padding unit sets the calculated average value pixel as each pixel in the restored subregion (non-ROI). Note that pixels adjacent to a subregion where encoded data was not transmitted are pixels in the ROI. Furthermore, pixels adjacent to a subregion where encoded data was not transmitted are adjacent pixels in the ROI adjacent to that subregion when that subregion is restored.

[0071] The region expansion unit sets the pixel value of the decoded picture (the pixel value of the said subregion) as the pixel value of the subregion whose magnification is not 0 or 1.0, and further expands the said subregion by affine transformation based on the magnification, and returns the position of the subregion whose magnification is not 0 or 1.0 to its original position (original coordinate position).

[0072] In this way, the picture is restored, and a picture based on the restored picture is output.

[0073] As described above, when the RTG processing units 102 and 204 perform RTG processing on a YUV4:4:4 format picture, the amount of processing required for RTG processing increases. Therefore, the following methods can be used to reduce the amount of processing required.

[0074] The RTG processing units 102 and 204 can perform RTG processing on a YUV4:2:0 format picture by defining the division points described above for each color component. Figure 7 is an explanatory diagram showing an example of processing by the RTG processing unit.

[0075] When performing RTG processing on a YUV4:2:0 format picture, the division point after forward transformation of the luminance component (Y component) is the same as the division point after forward transformation when performing RTG processing on a YUV4:4:4 format picture.

[0076] The RTG processing units 102 and 204 perform RTG processing on a YUV4:2:0 picture and set the coordinate values ​​of the division points for the color difference components (U and V components) to the values ​​obtained by right-shifting by one bit the coordinate values ​​of the division points when performing RTG processing on a YUV4:4:4 picture (equivalent to dividing the value by 1 / 2).

[0077] By using the coordinate values ​​of the division points defined as described above, the RTG processing units 102 and 204 can perform RTG processing on YUV4:2:0 format pictures. Therefore, the amount of processing required for RTG processing is reduced.

[0078] However, when division points are defined for each color component as described above, the magnification of a subregion related to the color difference component (a subregion of the color difference component) may differ from (not be the same as) the magnification of a subregion related to the luminance component (a subregion of the luminance component). Note that the same magnification for the luminance component and the same magnification for the color difference component means that when a transformation (e.g., affine transformation) is applied, the ratio of the size before and after the transformation (width and height, respectively) is the same for the subregion of the luminance component and the subregion of the color difference component. Note that the ratio of the size before and after the transformation is the magnification.

[0079] Figure 8 is a schematic diagram illustrating the RTG process. The left side of Figure 8 shows an example of coordinate values ​​before and after the forward transformation of the luminance component. In the example shown in Figure 8, the magnification of a subregion of the luminance component, i.e., the magnification or reduction ratio in the affine transformation, is 1.0. The right side of Figure 8 shows an example of coordinate values ​​before and after the forward transformation of the chrominance component. In the example shown in Figure 8, if the coordinate value after being halved is not an integer, the integer with the decimal part truncated is used as the coordinate value.

[0080] In the example shown in Figure 8, the size (width and height) of the subregion relating to the chromatic difference component (coordinate value × (1 / 2)) of the corresponding forward-transformed luminance component does not match the size of the subregion relating to the chromatic difference component (coordinate value × (1 / 2)) of the corresponding pre-forward-transformed luminance component. In other words, the magnification of the subregion relating to the chromatic difference component (magnification or reduction ratio in affine transformation) is different from the magnification of the subregion relating to the luminance component (1.0 in this embodiment). For this reason, the transformation unit 1024 shown in Figure 3 and the transformation unit 2044 shown in Figure 6 are required to perform scaling (magnification or reduction) that would not normally need to be performed. That is, the transformation units 1024 and 2044 are required to perform redundant processing. As a result, the processing load of the RTG process increases. Furthermore, if the RTG process is performed with a different magnification of the subregion relating to the chromatic difference component than the magnification of the subregion relating to the luminance component, the relationship between the position of the chromatic difference component and the position of the corresponding luminance component may deviate from the original positional relationship. In other words, a phase shift may occur. When a phase shift occurs, the quality of the video reproduced by the video decoder deteriorates.

[0081] Embodiment 1. The video encoder of this embodiment is configured in the same way as the video encoder 100 shown in Figure 1. However, the configuration and operation of the RTG processing unit are different from those in the RTG processing unit 102 shown in Figure 1.

[0082] Figure 9 is a block diagram showing an example of the configuration of the RTG processing unit in the video encoder of this embodiment. The RTG processing unit 102A shown in Figure 9 includes an ROI information acquisition unit 1021, a partial region definition unit 1022, a coordinate correction unit 1025, a parameter determination unit 1023, and a conversion unit 1024. The configuration and operation of the ROI information acquisition unit 1021, the partial region definition unit 1022, the parameter determination unit 1023, and the conversion unit 1024 are the same as those in the RTG processing unit 102 shown in Figure 3.

[0083] In other words, the configuration of the video encoder in this embodiment corresponds to the configuration in the video encoder 100 shown in Figure 1, in which the RTG processing unit 102 is replaced by the RTG processing unit 102A.

[0084] The video decoder in this embodiment is configured similarly to the video decoder 200 shown in Figure 1. However, the configuration and operation of the RTG processing unit differ from those in the RTG processing unit 204 shown in Figure 2.

[0085] Figure 10 is a block diagram showing an example of the configuration and operation of the RTG processing unit in the video decoder of this embodiment. The RTG processing unit 204A shown in Figure 10 includes an ROI information acquisition unit 2041, a partial region restoration unit 2042, a coordinate correction unit 2045, a parameter acquisition unit 2043, and a conversion unit 2044. The configuration and operation of the ROI information acquisition unit 2041, the partial region restoration unit 2042, the parameter acquisition unit 2043, and the conversion unit 2044 are the same as those in the RTG processing unit 204 shown in Figure 6.

[0086] In other words, the configuration of the video decoder in this embodiment corresponds to the configuration in the video decoder 200 shown in Figure 1, in which the RTG processing unit 204 is replaced by the RTG processing unit 204A.

[0087] The processing of the coordinate correction units 1025 and 2045 will now be explained. Figure 11 is an explanatory diagram illustrating the processing by the coordinate correction units 1025 and 2045 in this embodiment. In this embodiment, the coordinate correction units 1025 and 2045 correct the coordinate values ​​of the division points supplied from the partial region definition unit 1022 or the partial region restoration unit 2042 as follows.

[0088] The left side of Figure 11 shows an example of coordinate values ​​before and after the forward transformation of the luminance component. In the example shown in Figure 11, the magnification of a subregion of the luminance component, i.e., the magnification or reduction ratio in the affine transformation, is 1.0. The right side of Figure 11 shows an example of coordinate values ​​before and after the forward transformation of the chrominance component.

[0089] The coordinate correction units 1025 and 2045 control the rounding direction of the coordinate values ​​so that, when the magnification of the subregion of the luminance component is 1.0, the size of the subregion relating to the chromatic difference component (coordinate value × (1 / 2)) of the corresponding forward-transformed luminance component matches the size of the subregion relating to the chromatic difference component (coordinate value × (1 / 2)) of the corresponding pre-forward-transformed luminance component. In the example shown in Figure 11, if the coordinate value that has been divided by 2 is not an integer, the coordinate correction units 1025 and 2045 correct it so that the coordinate value becomes an integer with the decimal part rounded up.

[0090] In the example shown in Figure 11, the coordinate correction units 1025 and 2045 select the rounding direction for the coordinate values, but depending on the size of the subregion after the forward transformation, they may also select the rounding direction.

[0091] In this embodiment, RTG processing is performed with the magnification of the partial region relating to the color difference component being the same as the magnification of the partial region relating to the luminance component, so the conversion units 1024 and 2044 do not need to perform redundant processing.

[0092] Figure 12 is a flowchart showing the operation of the RTG processing unit 102A in the video encoder 100.

[0093] The ROI information acquisition unit 1021 acquires ROI location information based on information that allows the ROI to be identified, for example, in the control information input to the video encoder 100 from an external source (step S101). The partial region definition unit 1022 defines the partial region as described above and determines the magnification of the partial region (step S102).

[0094] As described above, the coordinate correction unit 1025 corrects the coordinates of the partial region (step S103).

[0095] The parameter determination unit 1023 determines the parameters for the affine transformation for each sub-region (step S104). The transformation unit 1024 performs the affine transformation operation for each sub-region based on the parameters for the affine transformation (step S105).

[0096] Figure 13 is a flowchart showing the operation of the RTG processing unit 204A in the video decoder 200.

[0097] The ROI information acquisition unit 2041 acquires ROI information from the side information (step S201). The ROI information includes the position information of the partial region and the ROI reduction ratio (magnification).

[0098] The partial region reconstruction unit 2042 reconstructs the position of the partial region based on the ROI position information, the ROI scaling factor, and information representing the relationship (difference) between the coordinates of the partial region after the affine transformation (see Example (B) in Figure 6) and the original coordinates (see Example (A) in Figure 6), which are included in the side information (step S202). Specifically, the partial region reconstruction unit 2042 reconstructs the division points as illustrated in Example (A).

[0099] As described above, the coordinate correction unit 2045 corrects the coordinates of the partial region (step S203).

[0100] The parameter acquisition unit 2043 determines the parameters for the affine transformation of each sub-region based on the side information (step S204). The parameters to be determined are the amount of displacement (translation) and the magnification factor. The magnification factor is determined based on the magnification factor included in the side information.

[0101] The padding section undergoes an affine transformation for each sub-region (step S205).

[0102] In this embodiment, the video decoder 200 sets a subregion for each subregion where the magnification is set to 0 by the video encoder 100, and the subregion has a pixel that has the average value of the adjacent pixel values. However, it is also conceivable to calculate the pixel values ​​of the pixels set in each subregion all at once. For example, when calculating the pixel values ​​set in two non-ROIs (R1, R2) all at once, if the average value of adjacent pixels (groups of adjacent pixels) is used, one method is to sequentially extract adjacent pixels of the first non-ROI (R1) and adjacent pixels of the second non-ROI (R2), and then calculate the average value of all extracted pixels (average value of the pixel group).

[0103] When using such a method, it is assumed that an extraction unit is provided in place of the padding unit described above to extract adjacent pixels of each non-ROI (pixels adjacent to the non-ROI in the ROI). Then, a padding unit is provided after the conversion unit to calculate the average value of the extracted pixels and set the calculated average value to the non-ROI (R1, R2). The extracted adjacent pixels are pixels in the ROI.

[0104] In such a configuration, padding is performed only after the process of extracting adjacent pixels for all non-ROI areas is completed, resulting in a waiting time until padding is finished. In the embodiment described above, for each sub-region (non-ROI area), the process of extracting adjacent pixels and padding are performed in the padding unit. Therefore, the above waiting time does not occur.

[0105] Embodiment 2. The video encoder of this embodiment is configured in the same way as the video encoder 100 shown in Figure 1. However, the configuration and operation of the RTG processing unit are different from those in the RTG processing unit 102 shown in Figure 1.

[0106] Furthermore, the configuration of the RTG processing unit in the video encoder of this embodiment is the same as the configuration of the RTG processing unit 102A in the first embodiment shown in Figure 9. However, the processing by the coordinate correction unit 1025 in this embodiment is different from the processing by the coordinate correction unit in the first embodiment.

[0107] The video decoder in this embodiment is configured similarly to the video decoder 200 shown in Figure 1. However, the configuration and operation of the RTG processing unit differ from those in the RTG processing unit 204 shown in Figure 2.

[0108] Furthermore, the configuration of the RTG processing unit in the video decoder of this embodiment is the same as the configuration of the RTG processing unit 204A in the first embodiment shown in Figure 10. However, the processing by the coordinate correction unit 2045 in this embodiment is different from the processing by the coordinate correction unit in the first embodiment.

[0109] The processing of the coordinate correction units 1025 and 2045 in this embodiment will be described. Figure 14 is an explanatory diagram illustrating the processing by the coordinate correction units 1025 and 2045 in this embodiment. In this embodiment, the coordinate correction units 1025 and 2045 correct the coordinate values ​​of the division points supplied from the partial region definition unit 1022 or the partial region restoration unit 2042 as follows.

[0110] The left side of Figure 14 shows an example of the coordinate values ​​before and after the forward transformation of the luminance component. In the example shown in Figure 14, the magnification of the subregion of the luminance component, i.e., the magnification or reduction ratio in the affine transformation, is 1.0. The right side of Figure 14 shows an example of the coordinate values ​​before and after the forward transformation of the chrominance component. Note that in this embodiment, it is not essential that the magnification of the subregion of the luminance component is 1.0.

[0111] The coordinate correction units 1025 and 2045 correct the coordinate values ​​of a subregion so that the width and height of the subregion of the luminance component before the affine transformation are even numbers. The coordinate correction units 1025 and 2045 also correct the coordinate values ​​of a subregion so that the width and height of the subregion of the luminance component after the affine transformation are even numbers. In the example shown in Figure 14, the coordinate correction units 1025 and 2045 correct the coordinate value of the upper left (upper left corner) of the subregion of the luminance component from, for example, (115, 0) (see Figure 11) to (114, 0). Furthermore, the coordinate correction units 1025 and 2045 correct the coordinate value of the lower right subregion of the luminance component from, for example, (137, 0) (see Figure 11) to (138, 0). If the coordinate value of the top left is transformed to (116, 0) by the affine transformation, then naturally the coordinate correction units 1025 and 2045 will transform the coordinate value of the bottom right (bottom right corner) to (138, 20).

[0112] Furthermore, in order to correct (constrain) the coordinate value a to an even number, the coordinate correction units 1025 and 2045 perform, for example, the following calculation: (a >> 1) << 1. That is, the coordinate correction units 1025 and 2045 right-shift the coordinate value a by 1 bit to halve the value and truncate the fractional part, and then left-shift it by 1 bit to return it to the original value (however, the fractional part has been truncated).

[0113] The coordinate correction units 1025 and 2045 may, for example, perform the following operation to round up the fractional part, thereby restricting the coordinate value a to an even number: ((a+1)>>)<<1

[0114] In the example shown in Figure 14, with respect to the color difference component, the sub-region definition unit 1022 and the sub-region restoration unit 2042 output (57, 0) as the upper-left coordinate value of the sub-region of the color difference component and (68, 10) as the lower-right coordinate value. As shown in Figure 14, due to the affine transformation, the upper-left coordinate value of the sub-region of the color difference component becomes (58, 0) and the lower-right coordinate value becomes (68, 10).

[0115] Figure 15 is a flowchart showing the operation of the RTG processing unit in the video encoder 100 of this embodiment.

[0116] In step S103A, the coordinate correction unit 1025 restricts the width and height values ​​of the luminance component subregions before and after the affine transformation to even values, as described above. The processes in steps S101, S102, S104, and S105 are the same as those shown in Figure 12.

[0117] Figure 16 is a flowchart showing the operation of the RTG processing unit in the video decoder 200 of this embodiment.

[0118] In step S203A, the coordinate correction unit 2045 restricts the width and height values ​​of the luminance component subregions before and after the affine transformation to even values, as described above. The processes in steps S201, S202, S204, and S205 are the same as those shown in Figure 13.

[0119] In this embodiment, the size of the subregion of the chromatic difference component after affine transformation is the same as the size of the subregion before affine transformation. That is, in the example shown in Figure 14, the magnification of the subregion of the chromatic difference component is 1.0, just like the magnification of the subregion of the luminance component. In this embodiment, since the RTG processing is performed with the magnification of the subregion related to the chromatic difference component being the same as the magnification of the subregion related to the luminance component, the transformation units 1024 and 2044 do not need to perform redundant processing.

[0120] Furthermore, as in this embodiment, when the coordinate correction units 1025 and 2045 correct the coordinate values ​​of a partial region so that the width and height of the partial region of the luminance component are even numbers, the magnification of the partial region of the chromatic difference component can be made to match the magnification of the partial region of the luminance component even when the magnification of the partial region of the luminance component is not 1.0.

[0121] Embodiment 3. The video encoder of this embodiment is configured in the same way as the video encoder 100 shown in Figure 1. However, the configuration and operation of the RTG processing unit are different from those in the RTG processing unit 102 shown in Figure 1.

[0122] Furthermore, the configuration of the RTG processing unit in the video encoder of this embodiment is the same as the configuration of the RTG processing unit 102A in the first embodiment shown in Figure 9. However, the processing by the coordinate correction unit 1025 in this embodiment is different from the processing by the coordinate correction unit in the first embodiment.

[0123] The video decoder in this embodiment is configured similarly to the video decoder 200 shown in Figure 1. However, the configuration and operation of the RTG processing unit differ from those in the RTG processing unit 204 shown in Figure 2.

[0124] Furthermore, the configuration of the RTG processing unit in the video decoder of this embodiment is the same as the configuration of the RTG processing unit 204A in the first embodiment shown in Figure 10. However, the processing by the coordinate correction unit 2045 in this embodiment is different from the processing by the coordinate correction unit in the first embodiment.

[0125] The processing of the coordinate correction units 1025 and 2045 in this embodiment will be described. As in the description of this embodiment, Figure 14 will be referred to. In this embodiment, the coordinate correction units 1025 and 2045 correct the coordinate values ​​of the division points supplied from the partial region definition unit 1022 or the partial region restoration unit 2042 as follows.

[0126] In other words, the coordinate correction units 1025 and 2045 restrict the coordinate values ​​of the upper left and lower right of the subregion before and after the affine transformation to even numbers. To put it another way, the coordinate correction units 1025 and 2045 restrict the coordinate values ​​of each vertex to even numbers. In the example shown in Figure 14, the coordinate correction units 1025 and 2045 correct the upper left coordinate value of the subregion of the luminance component before the affine transformation from, for example, (115, 0) (see Figure 11) to (114, 0). Also, the coordinate correction units 1025 and 2045 correct the lower right coordinate value of the subregion of the luminance component after the affine transformation from, for example, (137, 0) (see Figure 11) to (138, 0). If the affine transformation results in the upper left coordinate value being transformed to (116, 0), then naturally, the coordinate correction units 1025 and 2045 transform the lower right (lower right corner) coordinate value to (138, 20).

[0127] Furthermore, the operation of the RTG processing unit in the video encoder 100 of this embodiment is the same as the operation in the second embodiment, except that the process in step S103A in the flowchart of Figure 15 is replaced with a process that restricts the coordinate values ​​of each vertex in the subregion before and after the affine transformation to even values.

[0128] Furthermore, the operation of the RTG processing unit in the video decoder 200 of this embodiment is the same as the operation in the second embodiment, except that the process in step S203A in the flowchart of Figure 16 is replaced with a process that restricts the coordinate values ​​of the upper left and lower right of the subregion before and after the affine transformation to even values.

[0129] In this embodiment, the size of the subregion of the chromatic difference component after affine transformation is the same as the size of the subregion before affine transformation. That is, in the example shown in Figure 14, the magnification of the subregion of the chromatic difference component is 1.0, just like the magnification of the subregion of the luminance component. In this embodiment, since the RTG processing is performed with the magnification of the subregion related to the chromatic difference component being the same as the magnification of the subregion related to the luminance component, the transformation units 1024 and 2044 do not need to perform redundant processing.

[0130] Furthermore, as in this embodiment, if the coordinate correction units 1025 and 2045 restrict the coordinate values ​​of the upper left and lower right of the subregion before the affine transformation to even values, the magnification of the subregion of the chromatic difference component can be made to match the magnification of the subregion of the chromatic difference component even when the magnification of the subregion of the luminance component is not 1.0.

[0131] Furthermore, the coordinate correction units 1025 and 2045 restricting the coordinate values ​​of each vertex before the affine transformation of the luminance component subregion to even values ​​is equivalent to restricting the width and height of the luminance component subregion before the affine transformation to even numbers. Similarly, the coordinate correction units 1025 and 2045 restricting the coordinate values ​​of each vertex after the affine transformation of the luminance component subregion to even values ​​is equivalent to restricting the width and height of the luminance component subregion after the affine transformation to even numbers.

[0132] Furthermore, restricting the coordinate values ​​of each vertex before the affine transformation of a subregion of the luminance component to even numbers can be achieved by restricting the coordinate values ​​of each vertex of the detected ROI to even numbers. When the coordinate values ​​of an ROI are even, the video encoder halves the x and y coordinates, width, and height of the top-left corner of each ROI before including them in the side information, and the video decoder doubles the values ​​read when restoring the side information to obtain the correct values. In this case, designing the syntax with the assumption of halving the values ​​allows for a reduction in the bit length required to represent the information of each ROI, thus reducing the amount of data in the bitstream.

[0133] [Modified Example] In the third embodiment, the coordinate correction units 1025 and 2045 correct the coordinate values ​​(x-coordinate and y-coordinate values) of the subregion so that the coordinate values ​​of the vertices of the subregion of the luminance component before the affine transformation are constrained to even values. Furthermore, the coordinate correction units 1025 and 2045 correct the coordinate values ​​of the subregion so that the coordinate values ​​of the vertices of the subregion of the luminance component after the affine transformation are constrained to even values.

[0134] However, the coordinate correction units 1025 and 2045 may correct the coordinate values ​​of the vertices of the luminance component subregion before and after the affine transformation (hereinafter simply referred to as the coordinate values ​​of the subregion) so that they are multiples of a predetermined even number. That is, the coordinate correction units 1025 and 2045 may restrict the coordinate values ​​of the luminance component subregion before and after the affine transformation to multiples of a predetermined even number. In other words, in this modified example, in the processing of step S103A in Figure 15 and the processing of step S203A in Figure 16, the coordinate correction units 1025 and 2045 restrict the coordinate values ​​of the subregion before and after the affine transformation to multiples of a predetermined even number.

[0135] The predetermined even value is, for example, a fixed value predetermined in the video encoder 100 and the video decoder 200 (the same value in both the video encoder 100 and the video decoder 200).

[0136] Alternatively, a predetermined even value may be set in advance in the video encoder 100 and included in the bitstream of side information to be transmitted to the video decoder 200. In this case, the coordinate correction unit 1025 restricts the coordinate values ​​of the subregion before and after the affine transformation to multiples of the predetermined even value and supplies the predetermined even value to the multiplexing unit 106. The multiplexing unit 106 creates a bitstream containing information that allows for the identification of the predetermined even value. In the video decoder 200, the demultiplexing unit 201 extracts information that allows for the identification of the predetermined even value from the bitstream and supplies the predetermined even value to the coordinate correction unit 2045. The coordinate correction unit 2045 restricts the coordinate values ​​of the subregion before and after the affine transformation to multiples of the predetermined even value.

[0137] The coordinate correction units 1025 and 2045 restrict the coordinate values ​​of sub-regions before and after the affine transformation to predetermined even values, thereby suppressing the fine division of the picture to be processed. As a result, processing efficiency can be further improved.

[0138] Furthermore, the predetermined even value for the x-axis coordinate may be the same as or different from the predetermined even value for the y-axis coordinate. Also, the predetermined even value for the coordinate before the affine transformation and the predetermined even value for the coordinate after the affine transformation may be the same as or different from each other.

[0139] Figure 17 is an explanatory diagram illustrating the effect of the modified example. Figure 17 shows two ROIs (ROI1, ROI2) as examples. The y-coordinate Pv[j] of the division point is also indicated. Figure 17(A) shows an example where the coordinate value of the division point is not restricted to an even number. Figure 17(B) shows an example where the coordinate value of the division point is restricted to an even number. Figure 17(C) shows an example where the coordinate value of the division point is restricted to a multiple of a predetermined even number, as in this modified example.

[0140] As shown in Figure 17, even in situations where the number of division points does not decrease when the coordinate values ​​of the division points are constrained to even numbers, as in the third embodiment described above, the number of division points is reduced in this modified example. In other words, the division of the picture to be processed into smaller parts is suppressed. To put it another way, the RTG processing units 102A and 204A can process multiple finely divided sub-regions as a single sub-region with a larger area. This reduces the number of times the pixel data to be processed is read into memory, thus improving processing efficiency. Therefore, the increase in the amount of processing required in the video encoding scheme can be further suppressed.

[0141] While each of the above embodiments can be implemented using hardware, it can also be realized using computer programs.

[0142] The information processing system shown in Figure 18 includes a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Multiple processors can also be used. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media.

[0143] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiment.

[0144] The processor 701 then performs processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoder 100 and video decoder 200 shown in the above embodiment.

[0145] For example, the processor 701 performs processing according to a video encoding program for realizing the functions of each block in the video encoder 100 shown in Figure 1, thereby realizing the functions of the video encoder 100. In addition, the processor 701 performs processing according to the program to realize the functions of the video encoder 100 including the RTG processing unit 102A shown in Figure 9 (in the video encoder 100 configured in which the RTG processing unit 102 is replaced by the RTG processing unit 102A).

[0146] Furthermore, for example, the processor 701 performs processing according to a video decoding program for realizing the functions of each block in the video decoder 200 shown in Figure 1, thereby realizing the functions of the video decoder 200. In addition, the processor 701 performs processing according to the program to realize the functions of the video decoder 200 including the RTG processing unit 204A shown in Figure 10 (in the video decoder 200 configured in which the RTG processing unit 204 is replaced by the RTG processing unit 204A).

[0147] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0148] Figure 19 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 19 (implemented by a video encoder 100 in this embodiment) includes: a first acquisition means 11 (implemented by a partial region definition unit 1022 and a parameter determination unit 1023 in this embodiment) for acquiring a first coordinate (for example, a coordinate before affine transformation) indicating the position of a partial region in a picture; a second acquisition means 12 (implemented by a coordinate correction unit 1025 in this embodiment) for acquiring a second coordinate by performing a process to make the coordinate value even for the first coordinate; a third acquisition means 13 for acquiring a third coordinate (for example, a coordinate after affine transformation) based on the partial region represented by the second coordinate; and a correction means 14 (implemented by a coordinate correction unit 1025 in this embodiment) for performing a process to make the coordinate value even for the third coordinate.

[0149] Figure 20 is a block diagram showing the main part of the video decoding device. The video decoding device 20 shown in Figure 20 (implemented by a video decoder 200 in this embodiment) includes: a first acquisition means 21 (implemented by a partial region restoration unit 2042 and a parameter acquisition unit 2043 in this embodiment) for acquiring a first coordinate (for example, a coordinate before affine transformation) indicating the position of a partial region in a picture; a second acquisition means 22 (implemented by a coordinate correction unit 2045 in this embodiment) for acquiring a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; a third acquisition means 23 for acquiring a third coordinate (for example, a coordinate after affine transformation) based on the partial region represented by the second coordinate; and a correction means 24 (implemented by a coordinate correction unit 2045 in this embodiment) for performing a process to make the coordinate value of the third coordinate an even number.

[0150] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0151] (Note 1) A video encoding method that defines one or more subregions from a picture to be processed, performs an affine transformation operation for each subregion, and corrects the width or height of the subregion of the luminance component before the affine transformation if the width and height of the subregion of the luminance component after the affine transformation are the same as the width and height of the subregion of the luminance component before the affine transformation, so that the width and height of the subregion of the chrominance component after the affine transformation are the same as the width and height of the subregion of the chrominance component before the affine transformation.

[0152] (Note 2) A video encoding method that defines one or more subregions from a picture to be processed, performs an affine transformation operation for each subregion, and restricts the width and height of the subregion of the luminance component before the affine transformation, and the width and height of the subregion of the luminance component after the affine transformation, to even numbers.

[0153] (Note 3) The video encoding method according to Note 2, wherein the coordinate values ​​of the upper left and lower right of a subregion of the luminance component before the affine transformation are restricted to even values, and the coordinate values ​​of the upper left and lower right of a subregion of the luminance component after the affine transformation are restricted to even values.

[0154] (Note 4) The video encoding method according to Note 2, wherein the coordinate values ​​of the vertices of the luminance component subregion before the affine transformation are restricted to multiples of a predetermined even number, and the coordinate values ​​of the vertices of the luminance component subregion after the affine transformation are restricted to multiples of a predetermined even number.

[0155] (Note 5) A video decoding method comprising: decoding a video bitstream to obtain a decoded picture; obtaining one or more subregions from the decoded picture; performing an affine transformation operation for each subregion; and correcting the width or height of the subregion of the luminance component before the affine transformation if the width and height of the subregion of the luminance component after the affine transformation are the same as the width and height of the subregion of the luminance component before the affine transformation, so that the width and height of the subregion of the chrominance component after the affine transformation are the same as the width and height of the subregion of the chrominance component before the affine transformation.

[0156] (Note 6) A video decoding method comprising: decoding a video bitstream to obtain a decoded picture; obtaining one or more subregions from the decoded picture; performing an affine transformation operation for each subregion; and restricting the width and height of the subregion of the luminance component before the affine transformation, and the width and height of the subregion of the luminance component after the affine transformation, to even numbers.

[0157] (Note 7) The video decoding method according to Note 6, wherein the coordinate values ​​of the upper left and lower right of the luminance component subregion before the affine transformation are restricted to even values, and the coordinate values ​​of the upper left and lower right of the luminance component subregion after the affine transformation are restricted to even values.

[0158] (Note 8) The video decoding method according to Note 6, wherein the coordinate values ​​of the vertices of the luminance component subregion before the affine transformation are restricted to multiples of a predetermined even number, and the coordinate values ​​of the vertices of the luminance component subregion after the affine transformation are restricted to multiples of a predetermined even number.

[0159] (Note 9) A video encoding device (implemented by a video encoder 100 in this embodiment) comprising: a sub-region definition means for defining one or more sub-regions from a picture to be processed (implemented by a sub-region definition unit 1022 in this embodiment); a conversion means for performing an affine transformation operation for each sub-region (implemented by a conversion unit 1024 in this embodiment); and a correction means for correcting the width or height of the sub-region of the luminance component before the affine transformation (implemented by a coordinate correction unit 1025 in this embodiment) so that if the width and height of the sub-region of the luminance component after the affine transformation are the same as the width and height of the sub-region of the luminance component before the affine transformation, the width and height of the sub-region of the chrominance component after the affine transformation are the same as the width and height of the sub-region of the chrominance component before the affine transformation.

[0160] (Note 10) A video encoding device comprising: sub-region definition means for defining one or more sub-regions from a picture to be processed; transformation means for performing an affine transformation operation for each sub-region; and constraint means for restricting the width and height of the sub-region of the luminance component before affine transformation, and the width and height of the sub-region of the luminance component after affine transformation, to even numbers (implemented by a coordinate correction unit 1025 in this embodiment).

[0161] (Note 11) A video decoding device (implemented by a video decoder 200 in this embodiment) comprising: decoding means for decoding a video bitstream and obtaining a decoded picture (implemented by an internal decoder 202 in this embodiment); acquisition means for obtaining one or more subregions from the decoded picture (implemented by a subregion restoration unit 2042 in this embodiment); conversion means for performing an affine transformation operation for each subregion (implemented by a conversion unit 2044 in this embodiment); and correction means for correcting the width or height of the subregion of the luminance component before the affine transformation (implemented by a coordinate correction unit 2045 in this embodiment) so that the width and height of the subregion of the luminance component after the affine transformation are the same as the width and height of the subregion of the luminance component before the affine transformation if the width and height of the subregion of the luminance component after the affine transformation are the same as the width and height of the subregion of the luminance component before the affine transformation.

[0162] (Note 12) A video decoding device comprising: decoding means for decoding a video bitstream and obtaining a decoded picture; acquisition means for obtaining one or more subregions from the decoded picture; conversion means for performing an affine transformation operation for each subregion; constraint means for restricting the width and height of the subregion of the luminance component before affine transformation, and the width and height of the subregion of the luminance component after affine transformation, to even numbers; and constraint means for restricting the width and height of the subregion of the luminance component after affine transformation to even numbers (implemented by the coordinate correction unit 2045 in this embodiment).

[0163] (Note 13) A video encoding program that causes a computer to perform the following processes: defining one or more subregions from a picture to be processed; performing an affine transformation operation for each subregion; and correcting the width or height of the subregion of the luminance component before the affine transformation if the width and height of the subregion of the luminance component after the affine transformation are the same as the width and height of the subregion of the luminance component before the affine transformation, so that the width and height of the subregion of the chrominance component after the affine transformation are the same as the width and height of the subregion of the chrominance component before the affine transformation.

[0164] (Note 14) A video encoding program that causes a computer to perform the following processes: defining one or more subregions from a picture to be processed; performing an affine transformation operation for each subregion; and restricting the width and height of the subregions of the luminance components before the affine transformation, and the width and height of the subregions of the luminance components after the affine transformation, to be even numbers.

[0165] (Note 15) A video decoding program for causing a computer to perform the following processes: decoding a video bitstream and obtaining a decoded picture; obtaining one or more subregions from the decoded picture; performing an affine transformation operation for each subregion; and correcting the width or height of the subregion of the luminance component before the affine transformation if the width and height of the subregion of the luminance component after the affine transformation are the same as the width and height of the subregion of the luminance component before the affine transformation, so that the width and height of the subregion of the chrominance component after the affine transformation are the same as the width and height of the subregion of the chrominance component before the affine transformation.

[0166] (Note 16) A video decoding program that causes a computer to perform the following processes: decoding a video bitstream and obtaining a decoded picture; obtaining one or more subregions from the decoded picture; performing an affine transformation operation for each subregion; and restricting the width and height of the luminance component subregion before affine transformation, and the width and height of the luminance component subregion after affine transformation, to be even numbers.

[0167] Some or all of the configurations described in Appendices 3 and 4, which are dependent on Appendice 2 above, and some or all of the configurations described in Appendices 7 and 8, which are dependent on Appendice 6, can be made dependent on Appendices 9 to 10, Appendices 13 to 14, Appendices 11 to 12, and Appendices 15 to 16 in the same dependent relationships as Appendices 3 and 4 and Appendices 7 and 8. Furthermore, not limited to Appendices 9 to 10, Appendices 13 to 14, and Appendices 11 to 12, and Appendices 15 to 16, some or all of the configurations described as above-mentioned appendices can be made dependent on various hardware, software, various recording means for recording software, or systems, provided that the embodiments described above are not deviated from.

[0168] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0169] This application claims priority based on Japanese Patent Application No. 2024-187391, filed on 24 October 2024, and Japanese Patent Application No. 2025-003954, filed on 10 January 2025, and incorporates all of their disclosures herein.

[0170] 10 Video encoding device 11 First acquisition means 12 Second acquisition means 13 Third acquisition means 14 Correction means 20 Video decoding device 21 First acquisition means 22 Second acquisition means 23 Third acquisition means 24 Correction means 100 Video encoder 101 Time resampling unit 102, 102A RTG processing unit 103 Spatial resampling unit 104 Bit depth shift unit 105 Internal encoder 106 Multiplexer 200 Video decoder 201 Demultiplexer 202 Internal decoder 203 Spatial resampling unit 204, 204A RTG processing unit 205 Time resampling unit 206 Bit depth shift unit 701 Processor 702 Program memory 703, 704 Storage medium 1021 ROI information acquisition unit 1022 Partial region definition unit 1023 Parameter determination unit 1024 Transformation unit 1025 Coordinate correction unit 2041 ROI information acquisition unit 2042 Partial region restoration unit 2043 Parameter acquisition unit 2044 Transformation unit 2045 Coordinate correction unit

Claims

1. A video decoding method comprising: obtaining a first coordinate indicating the position of a partial region in a picture; obtaining a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; obtaining a third coordinate based on the partial region represented by the second coordinate; and performing a process to make the coordinate value of the third coordinate an even number.

2. The video decoding method according to claim 1, wherein the first coordinate, the second coordinate, and the third coordinate indicate the position of the upper left of the partial region.

3. The video decoding method according to claim 1, wherein the process of making the coordinate value an even number includes shifting the coordinate value to the right by one bit and then shifting it to the left by one bit.

4. The video decoding method according to claim 3, wherein, when processing to make the value of the first coordinate an even number, 1 is added to the first coordinate and then the value of the first coordinate is right-shifted by 1 bit.

5. The video decoding method according to claim 1, wherein the third coordinate indicates the position of the partial region after applying a process that changes at least one of the size or position of the partial region represented by the second coordinate.

6. A video encoding method comprising: obtaining a first coordinate indicating the position of a sub-region in a picture; obtaining a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; obtaining a third coordinate based on the sub-region represented by the second coordinate; and performing a process to make the coordinate value of the third coordinate an even number.

7. The video encoding method according to claim 6, wherein the first coordinate, the second coordinate, and the third coordinate indicate the position of the upper left of the partial region.

8. The video encoding method according to claim 6, wherein the process of making the coordinate value an even number includes shifting the coordinate value to the right by one bit and then shifting it to the left by one bit.

9. The video encoding method according to claim 8, wherein, when processing to make the value of the first coordinate an even number, 1 is added to the first coordinate and then the value of the first coordinate is right-shifted by 1 bit.

10. The video encoding method according to claim 6, wherein the third coordinate indicates the position of the partial region after applying a process that changes at least one of the size or position of the partial region represented by the second coordinate.

11. An image decoding device comprising: a first acquisition means for acquiring a first coordinate indicating the position of a partial region in a picture; a second acquisition means for acquiring a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; a third acquisition means for acquiring a third coordinate based on the partial region represented by the second coordinate; and a correction means for performing a process to make the coordinate value of the third coordinate an even number.

12. The video decoding device according to claim 11, wherein the first coordinate, the second coordinate, and the third coordinate indicate the upper left position of the partial region.

13. The video decoding apparatus according to claim 11, wherein the correction means performs a process of shifting the coordinate value to the right by one bit and then shifting it to the left by one bit, as a process to make the coordinate value an even number.

14. The video decoding device according to claim 13, wherein the correction means adds 1 to the first coordinate and then right-shifts the value of the first coordinate by 1 bit to make the value of the coordinate even with respect to the first coordinate.

15. The video decoding device according to claim 11, wherein the third coordinate indicates the position of the partial region after applying a process that changes at least one of the size or position of the partial region represented by the second coordinate.

16. A video encoding device comprising: a first acquisition means for acquiring a first coordinate indicating the position of a partial region in a picture; a second acquisition means for acquiring a second coordinate by performing a process to make the coordinate value of the first coordinate an even number; a third acquisition means for acquiring a third coordinate based on the partial region represented by the second coordinate; and a correction means for performing a process to make the coordinate value of the third coordinate an even number.

17. The video encoding apparatus according to claim 16, wherein the first coordinate, the second coordinate, and the third coordinate indicate the position of the upper left of the partial region.

18. The video encoding apparatus according to claim 16, wherein the correction means performs a process of shifting the coordinate value to the right by one bit and then shifting it to the left by one bit, as a process to make the coordinate value an even number.

19. The video encoding apparatus according to claim 18, wherein the correction means makes the value of the coordinate an even number with respect to the first coordinate by adding 1 to the first coordinate and then right-shifting the value of the first coordinate by 1 bit.

20. The video encoding apparatus according to claim 16, wherein the third coordinate indicates the position of the partial region after applying a process that changes at least one of the size or position of the partial region represented by the second coordinate.

Images