Video coding device, video decoding device, video coding method, and video decoding method

Abstract

This video coding device: defines a plurality of division points in the horizontal direction and sets a plurality of division points in the vertical direction in a processing target picture; defines a plurality of partial regions determined by straight lines extending from the division points; when the distance between division points adjacent to each other in the horizontal direction is shorter than a prescribed value, combines partial regions adjacent to each other in the horizontal direction by removing one of the division points; when the distance between division points adjacent to each other in the vertical direction is shorter than a prescribed value, combines partial regions adjacent to each other in the vertical direction by removing one of the division points; and executes affine transformation calculation for each of the partial regions.

Description

Video Encoding Device, Video Decoding Device, Video Encoding Method, and Video Decoding Method

[0001] The present disclosure relates to a video encoding device, a video decoding device, a video encoding method, and a video decoding method.

[0002] In order to efficiently transmit or record video, a video encoding device that generates an encoded representation (hereinafter referred to as a bitstream) of an input video, and a video decoding device that decodes the bitstream to generate a decoded video are used.

[0003] As standards for video encoding methods, there are H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High-Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29. In addition, as recent video encoding technologies, the technology described in Non-Patent Document 1 can be cited.

[0004] In these video encoding methods, generally, irreversible processing is executed in the video encoding device. By using irreversible processing, the data amount of the generated bitstream is suppressed. However, the quality of the video (decoded video) decoded by the video decoding device deteriorates compared to the quality of the video (input video) input to the video encoding device.

[0005] Various methods have been developed for applications for human viewing, including the above video encoding methods. Generally, the performance of these methods is evaluated from the perspective of how much the decoded video deteriorates with respect to the quality of the input video in terms of the quality of the video decoded from bitstreams of the same data amount generated according to the processing of each method. <已修正,原内容重复,推测有误,按照逻辑补充为]]

[0006] In recent years, not only for human viewing, but also the scenes where video is used for machine processing are increasing. For example, as a process executed by a machine, there is an object detection process of detecting a person or the like included in the video. [[ID=已修正,原内容重复,推测有误,按照逻辑补充为]]

[0007] Development of video encoding methods intended for machine processing is underway. For example, ISO / IEC / SC29 is working on standardizing a video encoding method for machines called Video for Coding Machines (VCM).

[0008] The performance of a video encoding scheme for machines is generally evaluated by the processing performance performed by the machine when using video decoded from a bitstream of a similar amount of data. Therefore, even if the quality of the decoded video is significantly degraded compared to the quality of the input video, the encoding scheme is considered to have good performance if the degree of degradation when using the decoded video is small compared to when using the input video. In other words, the difference in quality between the input video and the decoded video is not considered a problem. For this reason, video encoding devices that perform video encoding for machines may apply pre-processing that has not been used in schemes intended for human viewing, in order to improve the performance of the encoding scheme.

[0009] One preprocessing step to improve the performance of an encoding scheme is to replace the pixel values ​​of areas outside the region of interest in the input video, i.e., non-interest regions, with arbitrary pixel values. The region of interest is the area in the video that should be emphasized (important area). Hereafter, the region of interest may be referred to as ROI (Region of Interest). Also, objects such as people that should be emphasized in the region of interest will simply be referred to as objects. Non-Patent Document 2 discloses an example of preprocessing for machine-oriented video encoding, which includes the process of replacing the pixel values ​​of non-interest regions with arbitrary pixel values. Non-Patent Document 2 also shows an example of postprocessing when encoding and decoding video for machines.

[0010] Hereinafter, the process of reducing the amount of encoded data based on the location and size of the region of interest is referred to as ROI processing or RTG processing (retargeting processing). As ROI processing (RTG processing), for example, pixel values ​​in the region of non-interest are replaced with arbitrary pixel values ​​(for example, pixel values ​​corresponding to gray). Furthermore, the method disclosed in Non-Patent Document 3 utilizes a process that divides the picture to be processed into one or more rectangular regions based on the location of the region of interest and deletes the rectangular region corresponding to the region of non-interest. When this process is applied, the region of interest in the picture moves to the upper left.

[0011] Furthermore, in a method that allows referencing the H. 274 / VSEI standard, the coordinates of the region of interest detected within the picture can be transmitted as auxiliary information called annotated_regions, as defined in the H. 274 / VSEI standard.

[0012] "Algorithm description of Enhanced Compression Model 9 (ECM 9)", JVET-AD2025, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)", JVET-AD2030, JVET of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 30th Meeting, Antalya, TR, 21-28 April 2023"Region-of-Interest-Based Video Coding for Machines", IEEE International Conference on Multimedia and Expo Workshops, 2024

[0013] As mentioned above, ROI processing is a process that reduces the amount of encoded data, thereby reducing the amount of data transmitted from the video encoder to the video decoder. However, if the number of ROIs in a picture is large, the number of samples to be processed increases. As the number of samples increases, if processing is applied independently to each sample, the processing overhead for calling that processing increases as a proportion of the total processing amount. In that case, the processing efficiency of ROI processing decreases.

[0014] The purpose of this disclosure is to provide a video encoding device, a video decoding device, a video encoding method, a video decoding method, a video encoding program, and a video decoding program that can suppress a decrease in processing efficiency in the video encoding scheme.

[0015] The video encoding method based on this disclosure defines multiple division points in the horizontal direction and multiple division points in the vertical direction in the picture to be processed, defines multiple subregions defined by straight lines extending from the division points, combines horizontally adjacent subregions by removing one of the division points when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, combines vertically adjacent subregions by removing one of the division points when the distance between adjacent division points in the vertical direction is shorter than a predetermined value, and performs an affine transformation operation for each subregion.

[0016] The video decoding method based on this disclosure decodes a video bitstream to obtain a decoded picture, obtains multiple subregions from the decoded picture in which multiple division points are defined horizontally and multiple division points are defined vertically and defined by straight lines extending from the division points, combines horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, combines vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than a predetermined value, and performs an affine transformation operation for each subregion.

[0017] The video encoding apparatus based on this disclosure includes: sub-region definition means that defines a plurality of sub-regions defined by straight lines extending from a division point, by defining a plurality of division points in the horizontal direction and setting a plurality of division points in the vertical direction in a picture to be processed; division point removal means that combines horizontally adjacent sub-regions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combines vertically adjacent sub-regions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than a predetermined value; and transformation means that performs an affine transformation operation for each sub-region.

[0018] The video decoding device according to this disclosure includes decoding means for decoding a video bitstream and obtaining a decoded picture; acquisition means for obtaining a plurality of subregions from the decoded picture, each subregion defined by a plurality of division points in the horizontal direction and a plurality of division points in the vertical direction, and defined by straight lines extending from the division points; division point removal means for joining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and joining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than a predetermined value; and transformation means for performing an affine transformation operation for each subregion.

[0019] The video encoding program based on this disclosure causes a computer to perform the following processes: define multiple division points in the horizontal direction and set multiple division points in the vertical direction in a picture to be processed, and define multiple subregions defined by straight lines extending from the division points; combine horizontally adjacent subregions by removing one of the division points when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, combine vertically adjacent subregions by removing one of the division points when the distance between adjacent division points in the vertical direction is shorter than a predetermined value; and perform an affine transformation operation for each subregion.

[0020] The video decoding program based on this disclosure causes a computer to perform the following processes: decoding a video bitstream and obtaining a decoded picture; obtaining from the decoded picture multiple subregions defined by multiple division points in the horizontal direction and multiple division points in the vertical direction, and defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than a predetermined value; and performing an affine transformation operation for each subregion.

[0021] According to this disclosure, a video decoding device, a video encoding method, a video decoding method, a video encoding program, and a video decoding program are provided that can suppress an increase in the processing load in a video encoding scheme.

[0022] This is a block diagram illustrating embodiments of a video encoder and a video decoder. This is an explanatory diagram illustrating RTG processing. This is an explanatory diagram illustrating RTG processing. This is an explanatory diagram illustrating RTG processing. This is an explanatory diagram illustrating RTG processing. This is an explanatory diagram illustrating the problems of RTG processing. This is an explanatory diagram illustrating RTG processing in an embodiment. This is an explanatory diagram illustrating RTG processing in an embodiment. This is a block diagram illustrating an example configuration of the RTG processing unit in a video encoder. This is a block diagram illustrating an example configuration of the RTG processing unit in a video decoder. This is a flowchart illustrating an example of operation of a video encoder. This is a flowchart illustrating an example of operation of a video decoder. This is an explanatory diagram illustrating a modified version of RTG processing. This is an explanatory diagram illustrating the problems of RTG processing. This is an explanatory diagram illustrating RTG processing in an embodiment. This is an explanatory diagram illustrating RTG processing in an embodiment. This is a block diagram illustrating embodiments of a video encoder and a video decoder. This is a block diagram illustrating an example configuration of the RTG processing unit in a video encoder. This is a block diagram illustrating an example configuration of the RTG processing unit in a video decoder. This is a flowchart illustrating an example of operation of a video encoder. This is a flowchart illustrating an example of operation of a video decoder. This is an explanatory diagram illustrating a modified version of RTG processing. This is a block diagram showing an example of the configuration of an information processing system. This is a block diagram showing the main parts of a video encoding device. This is a block diagram showing the main parts of a video decoding device.

[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0024] Embodiment 1. Figure 1 is a block diagram showing an example of a video encoder 100 and a video decoder 200 suitable for a video encoding scheme for machines. Specifically, it is a block diagram showing an example of the configuration of a video encoder 100 that performs pre-processing and a video decoder 200 that performs post-processing.

[0025] The video encoder 100 includes a time resampling unit 101, an RTG processing unit (retargeting unit) 102, a spatial resampling unit 103, a bit depth shift unit 104, an internal encoder 105, and a multiplexer 106.

[0026] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectional flow. This is also true for other block diagrams.

[0027] The time resampling unit 101 performs time resampling on the input video (input picture). The time resampling performed by the time resampling unit 101 is, for example, frame decimation. The RTG processing unit 102 performs RTG processing on the frame decimated input video. The spatial resampling unit 103 applies spatial scalability to the output of the RTG processing unit 102. The spatial scalability performed by the spatial resampling unit 103 is, for example, picture reduction. The bit depth shifting unit 104 applies bit depth truncation to the output of the spatial resampling unit 103. The bit depth truncation performed by the bit depth shifting unit 104 is, for example, a 1-bit right shift of the pixel value.

[0028] The internal encoder 105 encodes the picture supplied from the bit depth shift unit 104 using a predetermined video encoding scheme. As the predetermined video encoding scheme, for example, an encoding scheme based on H.266 / VVC can be used. As an example, the predetermined video encoding scheme can be VVC with End2End added.

[0029] The multiplexer 106 outputs a bitstream that is a multiplexed bitstream of the video bitstream supplied from the internal encoder 105 and the bitstream of side information as auxiliary information.

[0030] The video decoder 200 includes a demultiplexer 201, an internal decoder 202, a spatial resampling unit 203, an RTG processing unit 204, a temporal resampling unit 205, and a bit depth shifting unit 206.

[0031] The demultiplexer 201 demultiplexes the bitstream input to the video decoder 200 to obtain the video bitstream and the side information bitstream. The demultiplexer 201 supplies the video bitstream to the internal decoder 202. The demultiplexer 201 supplies the side information bitstream to the RTG processing unit 204, etc.

[0032] The internal decoder 202 decodes the video bitstream and supplies the decoded picture (decoded video frame) to the spatial resampling unit 203.

[0033] The spatial resampling unit 203 applies spatial scalability to the decoded picture. The spatial scalability performed by the spatial resampling unit 203 is, for example, picture scaling. The RTG processing unit 204 performs RTG processing on the output of the spatial resampling unit 203. The temporal resampling unit 205 performs temporal resampling on the output of the RTG processing unit 204. The temporal resampling performed by the temporal resampling unit 205 is, for example, frame interpolation. The bit depth shift unit 206 reconstructs the image from the output of the temporal resampling unit 205. The bit depth shift unit 206 performs, for example, a 1-bit left shift of the pixel value.

[0034] Note that YUV4:2:0 indicates that signals in the YUV color space are encoded in YUV4:2:0 format.

[0035] Figures 2A and 2B are explanatory diagrams for illustrating the RTG process. Figure 2A shows an example of the presence of ROIs in a picture indicated by the outer rectangle. Figure 2A illustrates the presence of five ROIs (ROI1 to ROI5).

[0036] In RTG processing, horizontal and vertical division points are defined from the x and y coordinate values ​​of the vertices of each ROI. The picture to be processed is divided into a grid at the defined division points. Then, an affine transformation (scaling or shrinking, and translation) is applied to each of these division regions. An example of affine transformation processing is shown in Figure 6. Note that in Figures 2A and 2B, the horizontal direction (x direction) is defined as the horizontal direction, and the vertical direction (y direction) is defined as the vertical direction.

[0037] Figure 2B shows an example of setting sub-regions based on the location of an ROI. Sub-regions are defined by division points. That is, division points are determined so that each ROI is divided at the boundary of its region. For example, the coordinate values ​​of the horizontal division points are selected without overlap from the x-coordinates of the top-left and bottom-right points of each ROI. Similarly, the vertical division points are selected without overlap from the y-coordinates of the top-left and bottom-right points of each ROI.

[0038] As described above, in Figure 2A, the horizontal direction is defined as the x-direction (horizontal direction), and the vertical direction is defined as the y-direction (vertical direction). In the example shown in Figure 2A, 12 division points are defined in the x-direction, and 10 division points are defined in the y-direction. Note that the left and right edges, as well as the top and bottom edges of the picture, are also included as division points.

[0039] Figure 2B shows subregions based on the division points exemplified in Figure 2A. A subregion is a rectangular area enclosed by a line extending vertically from a division point in the x-direction and a division point extending horizontally from a division point in the y-direction. In the example shown in Figure 2B, 11 × 9 = 99 subregions are generated.

[0040] In typical RTG processing, an affine transformation is applied to each of the 99 subregions.

[0041] A reduction ratio (magnification) is determined for each ROI, and based on the ROI magnification, the horizontal and vertical magnifications of the regions between each division point, i.e., the horizontal and vertical magnifications of the divided regions, are determined. Hereinafter, the horizontal and vertical magnifications of the regions between division points may simply be referred to as the magnifications between division points.

[0042] As a method for determining the magnification of the ROI, for example, there is a method of setting the magnification of the ROI with a size equal to or greater than a predetermined value to be greater than 0 and less than 1.0 (0 to 1.0), and setting the magnification of the ROI with a size less than the predetermined value to 1.0. Thereafter, the magnification between the vertical division points is set to the maximum magnification among the magnifications of the ROIs located between each division point. When there is no ROI, 0 is set. Similarly, the magnification between the vertical division points is determined from the magnifications of the ROIs between each division point. The magnifications between the horizontal and vertical division points are represented by one-dimensional arrays Sh and Sv, respectively.

[0043] FIGS. 3A and 3B are explanatory diagrams for explaining the RTG process, and an example of the magnification is shown in FIG. 3B. The content of FIG. 3A is the same as the content of FIG. 2A.

[0044] The parameters of the affine transformation in the video encoder 100 can be expressed as follows.

[0045] The coordinates and scaling factors of the division points before the affine transformation are expressed as follows. Note that the affine transformation is sometimes simply referred to as "transformation". i and j are non-negative integers. The number of division points in the horizontal and vertical directions are represented by i and j, respectively, and the width and height of the picture before the transformation are represented by W and H. In this case, if the left and right edges of the picture before the transformation are to be included, set Ph[0]=0 and Ph[I-1]=W. Similarly, if the top and bottom edges of the picture before the transformation are to be included, set Pv[0]=0 and Pv[J-1]=H. Note that division points do not always have to include two division points. For example, if vertices of the ROI have the same coordinate values, they can be treated as a single division point. (1) The i-th division point in the horizontal direction before transformation: Ph[i] (Identified from the ROI position information.) (2) The j-th division point in the vertical direction before transformation: Pv[j] (Identified from the ROI position information.) (3) Horizontal scaling factor between division point Ph[i-1] and division point Ph[i] (i>0): Sh[i-1] (Determined by the RTG processing unit 102.) (4) Vertical scaling factor between division point Pv[j-1] and division point Pv[j] (j>0): Sv[j-1] (Determined by the RTG processing unit 102.)

[0046] The coordinates of the horizontal division points after transformation are calculated in the RTG processing unit 102 as follows: (5-1) i=0: Ph'[0] = 0 (5-2) i>0: Ph'[i] = Ph'[i-1] + (Ph[i]-Ph[i-1])*Sh[i-1] The coordinates of the vertical division points after transformation are calculated in the RTG processing unit 102 as follows: (6-1) j=0: Pv'[0] = 0 (6-2) j>0: Pv'[j] = Pv'[j-1] + (Pv[j]-Pv[j-1])*Sv[j-1]

[0047] The parameters of the affine transformation representing the movement amount (translation amount) of the i-th division point in the horizontal direction and the movement amount of the j-th division point in the vertical direction, and the magnification of the i-th division point in the horizontal direction and the magnification of the j-th division point in the vertical direction are determined as follows in the RTG processing unit 102. (7) Movement amount in each direction: Ph[i] - Ph’[i], Pv[j] - Pv’[j] (8) Magnification in each direction: Sh[i + 1], Sv[j + 1]

[0048] Note that the RTG processing unit 102 supplies at least the coordinate information before the affine transformation of each ROI, that is, the original coordinate information (corresponding to the coordinates of the division points shown in Example (A)), and the information indicating the magnification of the ROI to the multiplexer 106. The multiplexer 106 transmits those information to the video decoder 200 as side information. The RTG processing unit 204 in the video decoder 200 can obtain the above (1) to (8) based on those information.

[0049] Specifically, the video decoder 200 can determine as follows (11) to (19).

[0050] The coordinates and magnification of the division points before the transformation are determined as follows. Note that the coordinates and magnification of the division points before the transformation can be obtained from the side information included in the bit stream from the video encoder 100. Also, hereinafter, the affine transformation in the video decoder 200 may be referred to as an inverse transformation. (11) The i-th division point in the horizontal direction before the inverse transformation: Ph’[i] (12) The j-th division point in the vertical direction before the inverse transformation: Pv’[j] (13) The magnification in the horizontal direction between the division point Ph[i - 1] and the division point Ph[i] (i > 0): 1 / Sh[i - 1] (14) The magnification in the vertical direction between the division point Pv[j - 1] and the division point Pv[j] (j > 0): 1 / Sv[j - 1] (15) The i-th division point in the horizontal direction after the inverse transformation: Ph[i] (determined by the RTG processing unit 204.) (16) The j-th division point in the vertical direction after the inverse transformation: Pv[j] (determined by the RTG processing unit 204.)

[0051] The parameters of the affine transformation, which represent the displacement of the i-th division point in the horizontal direction and the displacement of the j-th division point in the vertical direction, as well as the scaling factor of the i-th division point in the horizontal direction and the scaling factor of the j-th division point in the vertical direction, are expressed as follows:

[0052] If Sh[i]>0 and Sv[j]>0, then: (17) Amount of movement in each direction: Ph[i]-Ph'[i], Pv[j]-Pv'[j] (18) Magnification in each direction: Sh[i], Sv[j]

[0053] Otherwise (when either or both Sh[i] and Sv[j] are 0) (19) Padding with the mean values ​​in the ranges Ph'[i]-1 to Ph'[i+1] and Pv'[j]-1 to Pv'[j+1]

[0054] As the number of ROIs increases, the number of division points in each direction also increases, resulting in a finer division of the picture. In other words, the number of subregions increases. If affine transformations and encoding processes are performed for each subregion, an increase in the number of subregions increases the processing load and decreases processing efficiency.

[0055] Figure 4 is an explanatory diagram illustrating the challenges of the RTG processing described above. In Figure 4, the horizontal and vertical scaling factors of the six subregions enclosed by thick borders are the same. However, in the method described above, the six subregions are processed independently.

[0056] In this embodiment, the video encoder 100 treats the six subregions enclosed by the thick border as a single subregion and performs processing (including affine transformation).

[0057] In Figure 4, the six rightmost subregions with non-zero horizontal magnification are outlined with thick borders. However, the video encoder 100 can treat the other six subregions with non-zero horizontal magnification as a single subregion and perform processing accordingly.

[0058] Furthermore, although Figure 4 illustrates multiple vertically adjacent subregions, the video encoder 100 also treats multiple horizontally adjacent subregions as a single subregion if the horizontal magnification is the same, and performs processing (including affine transformation) on them.

[0059] In other words, the video encoder 100 removes the division point between adjacent subregions (corresponding to the subregions within the thick frame in Figure 4) if their vertical magnifications are the same in the vertical direction (vertical direction, i.e., the y-direction). In other words, when j > 0, the j-th division point is removed when the vertical magnification of the subregion formed by the (j-1)th division point and the j-th division point is the same as the vertical magnification of the subregion formed by the j-th division point and the (j+1)th division point. Note that j is an integer of 1 or more.

[0060] Furthermore, the video encoder 100 removes the division point between adjacent subregions if their horizontal magnifications are the same in the horizontal direction (i.e., the x-direction). In other words, when i > 0, the i-th division point is removed when the horizontal magnification of the subregion formed by the (i-1)th division point and the i-th division point is the same as the horizontal magnification of the subregion formed by the i-th division point and the (i+1)th division point. Note that i is an integer greater than or equal to 1.

[0061] By performing such processing, the video encoder 100 reduces the number of division points. As a result, the number of subregions decreases. In other words, the video encoder 100 can process multiple subregions simultaneously. Consequently, the video encoder 100 can suppress a decrease in processing efficiency.

[0062] Figures 5A and 5B are explanatory diagrams illustrating the RTG processing of an embodiment in which multiple sub-regions are processed simultaneously. The content of Figure 5A is the same as that of Figure 3B, but in Figure 5A, the straight lines extending from the division points that can be removed are shown as dashed lines. Figure 5B shows the sub-regions after the division points have been removed. The number of sub-regions exemplified in Figure 5B is reduced compared to the number of sub-regions exemplified in Figure 3B. Therefore, compared to the case where sub-regions are defined as exemplified in Figure 3B, the processing efficiency is improved in this embodiment.

[0063] Furthermore, by using the region outside each of the new subregions obtained through the merging process (the process of removing the division points mentioned above) as a reference region, an improvement in the quality of the decoded image can be expected.

[0064] Figure 6 is a block diagram showing an example configuration of the RTG processing unit 102 in the video encoder 100. In the example configuration shown in Figure 6, the RTG processing unit 102 includes an ROI information acquisition unit 1021, a partial region definition unit 1022, a parameter determination unit 1023, and a conversion unit 1024.

[0065] The ROI information acquisition unit 1021 acquires ROI position information based on information that allows for the identification of the ROI, which is included in the control information input to the video encoder 100 from an external source. For example, the coordinate information includes the x (horizontal direction) and y (vertical direction) coordinates of the upper left corner of the region, and the size of the region (size in the x direction and size in the y direction). The video encoder 100 may also include an object detection unit that detects objects from a picture, and the object detection unit may define the region containing an object as the ROI. As a method for detecting an object, for example, a neural network trained to detect a specific object may be used. In this case, the neural network takes, for example, the pixel values ​​of the input video picture as input and outputs the coordinates of the region determined to contain an object. The output coordinates are defined as the coordinates of the region of interest. The region of interest for each picture may be calculated only from the processing results of that picture, or it may be calculated based on the processing results of multiple pictures. If the ROI is calculated based on the processing results of multiple pictures, the ROI information acquisition unit 1021 buffers the pictures until a predetermined number of pictures are input to the RTG processing unit 204.

[0066] The sub-region definition unit 1022 sets the sub-regions as follows, for example.

[0067] The sub-region definition unit 1022 determines the division points in the picture based on the position information of the ROIs. Specifically, the sub-region definition unit 1022 defines the division points so that each ROI is divided at the boundary portion of the ROI's region. For example, the sub-region definition unit 1022 sets the coordinate values ​​of the horizontal division points to values ​​selected without overlap from the x-coordinates of the top-left and bottom-right points of each ROI. Similarly, the vertical division points are selected without overlap from the y-coordinates of the top-left and bottom-right points of each ROI. The sub-region definition unit 1022 then sets the arrays obtained by arranging the division point values ​​in each direction in ascending order to form one-dimensional arrays Ph and Pv, respectively.

[0068] The sub-region definition unit 1022 may create a coordinate list of division points. In that case, the sub-region definition unit 1022 first adds the x-coordinates of the rightmost and leftmost points of the picture to the coordinate list. Next, the sub-region definition unit 1022 adds the x-coordinates of the leftmost and rightmost points of each ROI to the coordinate list. However, the sub-region definition unit 1022 does not add the coordinates of division points that overlap with the coordinates of other division points to the coordinate list.

[0069] In Figure 6, example (A) illustrates that six division points are defined horizontally and five division points are defined vertically. The starting point (point with coordinate value 0) and ending point (point with coordinate value 400 or 300) are also included as division points. In this case, Ph[i] = {0, 100, 220, 300, 360, 400} and Pv[j] = {0, 50, 150, 250, 300}. Based on these division points, the regions in which the picture to be processed is divided into a grid are defined as sub-regions before transformation (before affine transformation). That is, the sub-region definition unit 1022 defines multiple division points horizontally and sets multiple division points vertically in the picture to be processed, and defines multiple sub-regions which are multiple rectangular regions defined by straight lines extending from the division points.

[0070] Furthermore, in Figure 6, Example (B) shows an example of a picture after ROI processing has been applied.

[0071] The partial region definition unit 1022 also performs partial region merging processing as described with reference to Figures 4, 5A, and 5B, but in examples (A) and (B) in Figure 6, the partial region merging processing is omitted.

[0072] In other words, examples (A) and (B) are illustrative examples for explaining affine transformations. Examples (A) and (B) do not reflect the process of joining subregions.

[0073] The partial region definition unit 1022 determines a magnification for each ROI and determines the magnification between each division point based on the ROI magnification. As described above, one method for determining the ROI magnification is to set the magnification of ROIs greater than or equal to a predetermined value to greater than 0 and less than 1.0 (0 to 1.0), and the magnification of ROIs less than a predetermined value to 1.0. Subsequently, the magnification between division points in the vertical direction (horizontal direction magnification) is set to the maximum magnification among the ROIs located between each division point. If no ROI exists, 0 is set. Similarly, the magnification between division points in the vertical direction is determined from the ROIs located between each division point. In example (A), the scaling factors Sh[i] and Sv[j] between the horizontal and vertical division points are: Sh[i] = {0.0, 0.8, 0.0, 1.0, 0.0} and Sv[j] = {0.0, 0.8, 1.0, 0.0}. Example (A) illustrates that the vertical scaling factor of one ROI (the ROI containing the larger object) is set to 0.8 or 1.0, and the horizontal scaling factor is set to 0.8. It also illustrates that the vertical scaling factor of the other ROI (the ROI containing the smaller object) is set to 1.0, and the horizontal scaling factor is set to 1.0.

[0074] The sub-region definition unit 1022 calculates the position of the division points after conversion from the division points before conversion and the magnification ratio between those division points. The calculation method is as described above. In example (A), Ph[i]'={0,0,96,96,156,156} and Pv[j]'={0,0,80,180,180}, and the width and height after conversion are 156 and 180, respectively.

[0075] The sub-region definition unit 1022 includes the scaling factor of the ROI in the side information, along with the coordinate information of each ROI before affine transformation, i.e., the original coordinate information (corresponding to the division points shown in example (A)). The sub-region definition unit 1022 also includes information that identifies the size of the picture before and after the transformation in the side information. In the example shown in Figure 6, the scaling factor (0.8, 0.8) (the former is the horizontal scaling factor, the latter is the vertical scaling factor) is included in the side information along with the coordinate values ​​(100, 50, 220, 150) (the first two numbers are the x and y coordinates of the upper left of the region, and the latter two numbers are the x and y coordinates of the lower right of the region). Furthermore, the scaling factor (0.8, 1.0) is included in the side information along with the coordinate values ​​(100, 150, 220, 250). In addition, the scaling factor (1.0, 1.0) is included in the side information along with the coordinate values ​​(300, 50, 360, 250).

[0076] In the set of subregions through which encoded data will be transmitted, subregions other than subregion C (see Figure 6) are regions that encompass all or part of one or more ROIs. Hereinafter, the upper right subregion C, which does not contain an object, may be referred to as a specific region. For example, the subregion definition unit 1022 replaces the pixel values ​​of the specific region with fixed values ​​(for example, pixel values ​​corresponding to gray). In other words, with respect to a subregion that does not contain an object (corresponding to a type of non-ROI), the video encoder 100 does not delete the subregion that does not contain an object if a subregion containing an object exists in the same row or column. The specific region is then treated in the same way as an ROI.

[0077] Furthermore, the partial region definition unit 1022 sets the magnification of partial regions that do not contain objects (non-ROI) to 0.

[0078] Note that a subregion with a reduction ratio (magnification) of 1.0 means that the subregion will not be reduced. A subregion with a magnification of 0 means that the subregion will be deleted.

[0079] After the sub-region definition unit 1022 determines the magnification ratio between division points, it performs a sub-region merging process as described with reference to Figures 4, 5A, and 5B to reduce the number of division points before transformation. If the sub-region definition unit 1022 has created a coordinate list of division points, it deletes the coordinates of the division points removed in the merging process from the coordinate list.

[0080] The parameter determination unit 1023 determines the parameters for the affine transformation for each subregion. That is, the parameter determination unit 1023 moves the subregion containing the ROI to the upper left of the picture and determines the parameters so that the subregion is reduced by the determined reduction ratio. The reduction ratio (magnification) of the subregion containing the ROI is transmitted to the video decoder 200 as side information about the parameters of the affine transformation, for example. The method for determining the parameters of the affine transformation is as already described.

[0081] Then, the transformation unit 1024 performs the affine transformation operation for each sub-region based on the parameters of the affine transformation, as follows.

[0082] The transformation unit 1024 moves (translates) the subregion with a magnification of 1.0 by affine transformation. In the example shown in Figure 6, the other ROI (the ROI containing the smaller object) is translated.

[0083] The transformation unit 1024 shrinks and moves (translates) the subregions whose magnification is not 0 or 1.0 by affine transformation. In the example shown in Figure 6, one ROI (the ROI containing the larger object) is shrunk and translated.

[0084] In this way, a picture with ROI processing applied, as shown in example (B) in Figure 6, is created.

[0085] The video encoder 100 does not transmit the encoded data of the picture shown in example (A), but rather transmits the encoded data of the picture which is a set of subregions shown in example (B) (four subregions in example (B)). In other words, the video encoder 100 does not transmit the encoded data of subregions that do not include some ROIs.

[0086] Furthermore, the RTG processing unit 102 in the video encoder 100 also includes in the side information the size of the set of subregions after performing ROI processing including affine transformation (see example (B) in Figure 6), that is, the size of the set of subregions to which the encoded data is transmitted (since the coordinate value of the top left of the set is 0, it can be represented by the coordinate value of the bottom right), and information representing the relationship (difference) between the coordinates of the set and the original coordinates. The video encoder 100 may also include the number of ROIs (two in example (B)) in the side information.

[0087] Figure 7 is a block diagram showing an example of the configuration and operation of the RTG processing unit 204 in the video decoder 200. In the configuration example shown in Figure 7, the RTG processing unit 204 includes an ROI information acquisition unit 2041, a partial region restoration unit 2042, a parameter acquisition unit 2043, and a conversion unit 2044.

[0088] The ROI information acquisition unit 2041 acquires the ROI position information and ROI reduction ratio (magnification) from the side information obtained by demultiplexing, as well as the size of the picture before and after RTG processing in the video decoder 200.

[0089] The partial region reconstruction unit 2042 obtains division points from the size of the picture after RTG processing in the video encoder 100 and the ROI position information. Referring to the example shown in Figure 7, the partial region reconstruction unit 2042 reconstructs the division points shown in example (A). That is, the partial region reconstruction unit 2042 obtains multiple partial regions defined by multiple division points in the horizontal direction and multiple division points in the vertical direction, and defined by straight lines extending from the division points.

[0090] Furthermore, the partial region reconstruction unit 2042 performs the same process as the partial region merging process performed by the partial region definition unit 1022 in the video encoder 100 to reduce the number of division points. In other words, the partial region reconstruction unit 2042 reduces the number of partial regions.

[0091] The parameter acquisition unit 2043 determines the parameters for the affine transformation from the side information. The parameter acquisition unit 2043 moves the subregion including the ROI to the original coordinate position and determines the parameters so that the subregion is enlarged by the reciprocal of the determined magnification. If the magnification is 0, the parameter acquisition unit 2043 sets the enlargement ratio to 0.

[0092] The conversion unit 2044 includes a pixel copy unit, a padding unit, and a region expansion unit. The pixel copy unit sets the pixel value of the decoded picture (the pixel value of the said region) as the pixel value of a subregion with a magnification of 1.0. Then, based on the ROI position information, the pixel copy unit uses an affine transformation to return the ROI to its original position (the original coordinate position before the affine transformation was applied in the video encoder 100). The pixel value of the subregion is the pixel value of each pixel within the subregion. In other words, setting a pixel value in a subregion means filling the subregion with pixels that have that pixel value, or in other words, setting pixels that have that pixel value in the subregion. Translation can also be achieved, for example, by copying data (pixels) from one region in memory to another region.

[0093] The padding unit determines the pixel values ​​to be set in a sub-region where the magnification in any direction is 0, and sets the determined pixel values ​​in that sub-region. In other words, the conversion unit 2044 pads the sub-regions where encoded data was not transmitted from the video encoder 100 with the determined pixel values. The pixel values ​​to be set in a sub-region with a magnification of 0 are determined, for example, as follows.

[0094] The padding unit calculates statistical values ​​of pixels adjacent to a subregion where encoded data was not transmitted (for example, the values ​​of pixels surrounding a subregion with a width of 1 pixel). The following example uses the case where the average value is used as the statistic. When the magnification of the i-th subregion horizontally and the j-th subregion vertically is 0, the padding unit treats pixels in the pre-transformation picture where the x-coordinate is between Ph'[i]-1 and P'h[i+1], and the y-coordinate is between Pv'[j]-1 and Pv'[j+1], as adjacent pixels before transformation. The padding unit calculates the average value of adjacent pixels in all subregions where the magnification is 0. Then, the padding unit sets the calculated average value pixel as each pixel in the restored subregion (non-ROI). Note that pixels adjacent to a subregion where encoded data was not transmitted are pixels in the ROI. Furthermore, pixels adjacent to a subregion where encoded data was not transmitted are adjacent pixels in the ROI adjacent to that subregion when that subregion is restored.

[0095] The region expansion unit sets the pixel value of the decoded picture (the pixel value of the said subregion) as the pixel value of the subregion whose magnification is not 0 or 1.0, and further expands the said subregion by affine transformation based on the magnification, and returns the position of the subregion whose magnification is not 0 or 1.0 to its original position (original coordinate position).

[0096] In this way, the picture is restored, and a picture based on the restored picture is output.

[0097] In the video encoder 100, when the sub-region definition unit 1022 performs a sub-region merging process as described with reference to Figures 4, 5A, and 5B, a bitstream containing encoded data in which affine transform and encoding processes have been performed on a single sub-region obtained by merging multiple sub-regions, and a bitstream containing side information that allows the removed division points to be restored, are transmitted to the video decoder 200. Therefore, the video decoder 200 can process multiple sub-regions at once, similar to the video encoder 100. As a result, the video decoder 200 can suppress a decrease in processing efficiency. Although the explanation has been limited to the case where the processing applied to the sub-region is affine transform, similar effects can be obtained when padding or other image processing is applied to the sub-region.

[0098] Figure 8 is a flowchart showing the operation of the RTG processing unit 102 in the video encoder 100.

[0099] The ROI information acquisition unit 1021 acquires ROI location information based on information that allows the ROI to be identified, for example, in the control information input to the video encoder 100 from an external source (step S101). The partial region definition unit 1022 defines the partial region as described above and determines the magnification of the partial region (step S102).

[0100] Furthermore, the partial region definition unit 1022 performs the partial region merging process as described above (step S103).

[0101] The parameter determination unit 1023 determines the parameters for the affine transformation for each sub-region (step S104). The transformation unit 1024 performs the affine transformation operation for each sub-region based on the parameters for the affine transformation (step S105).

[0102] Figure 9 is a flowchart showing the operation of the RTG processing unit 204 in the video decoder 200.

[0103] The ROI information acquisition unit 2041 acquires ROI information from the side information (step S201). The ROI information includes at least the position information of the partial region and the ROI reduction ratio (magnification).

[0104] The partial region reconstruction unit 2042 reconstructs the position of the partial region based on the ROI position information, the ROI scaling factor, and information representing the relationship (difference) between the coordinates of the partial region after the affine transformation (see Example (B) in Figure 7) and the original coordinates (see Example (A) in Figure 7), which are included in the side information (step S202). Specifically, the partial region reconstruction unit 2042 reconstructs the division points as illustrated in Example (A) in Figure 7.

[0105] In step S202, the partial region restoration unit 2042 further reduces the division points using a process similar to that performed by the partial region definition unit 1022 in the video encoder 100.

[0106] The parameter acquisition unit 2043 determines the parameters for the affine transformation of each sub-region based on the side information (step S203). The parameters to be determined are the amount of displacement (translation) and the magnification factor. The magnification factor is determined based on the magnification factor included in the side information.

[0107] The padding section undergoes an affine transformation for each sub-region (step S204).

[0108] In this embodiment, the video decoder 200 sets a subregion for each subregion where the magnification is set to 0 by the video encoder 100, and the subregion has a pixel that has the average value of the adjacent pixel values. However, it is also conceivable to calculate the pixel values ​​of the pixels set in each subregion all at once. For example, when calculating the pixel values ​​set in two non-ROIs (R1, R2) all at once, if the average value of adjacent pixels (groups of adjacent pixels) is used, one method is to sequentially extract adjacent pixels of the first non-ROI (R1) and adjacent pixels of the second non-ROI (R2), and then calculate the average value of all extracted pixels (average value of the pixel group).

[0109] When using such a method, it is assumed that an extraction unit is provided in place of the padding unit described above to extract adjacent pixels of each non-ROI (pixels adjacent to the non-ROI in the ROI). Then, a padding unit is provided after the conversion unit to calculate the average value of the extracted pixels and set the calculated average value to the non-ROI (R1, R2). The extracted adjacent pixels are pixels in the ROI.

[0110] In such a configuration, padding is performed only after the process of extracting adjacent pixels for all non-ROI areas is completed, resulting in a waiting time until padding is finished. In the embodiment described above, for each sub-region (non-ROI area), the process of extracting adjacent pixels and padding are performed in the padding unit. Therefore, the above waiting time does not occur.

[0111] [Modified Version] In the video encoder 100 and video decoder 200, when the sub-region definition unit 1022 and the sub-region restoration unit 2042 perform a merging process, the sub-region containing the ROI and a specific region (sub-region C: see Figure 6) may be merged to form a single sub-region.

[0112] Figure 10 is an explanatory diagram illustrating a modified version of the RTG processing. In the example shown in Figure 10, the RTG processing (particularly the merging process described above) merges the subregion containing ROI2 with a specific region. That is, the merging process removes the division point between the subregion containing ROI2 and the specific region. If the decoding process is performed in this state, pixels outside the subregion containing ROI2 may be referenced. In that case, the image quality of the subregion containing ROI2 may deteriorate.

[0113] Therefore, in this modified example, the partial region reconstruction unit 2042 separates a single partial region formed by combining the partial region containing ROI2 and the specific region into the partial region containing ROI2 and the specific region. For example, the partial region reconstruction unit 2042 identifies the boundary between ROI2 and the specific region based on the ROI's position information. Then, it divides the partial region into two regions at that boundary. As a result, cross-boundary references can be prevented during the decoding process for ROI2.

[0114] Embodiment 2. Other issues will be described with reference to Figures 2A, 2B, 3A, 3B and 11. Assume that a partial region is defined and the magnification of the partial region is determined, as illustrated in Figures 2A, 2B, 3A and 3B.

[0115] As mentioned above, as the number of ROIs increases, the number of division points in each direction also increases, resulting in the picture being divided into smaller parts. In other words, the number of subregions increases. If encoding is performed for each subregion, an increase in the number of subregions increases the processing load and decreases processing efficiency.

[0116] Figure 11 is an explanatory diagram illustrating the challenges of RTG processing. In Figure 11, each rectangle represents a subregion. The region enclosed by the thick border is the region between the division points that define the subregion containing the ROI (the region that does not contain the ROI), but it contains multiple subregions. In the example shown in Figure 11, there are nine subregions. In typical RTG processing, an affine transformation is applied to each of the nine subregions.

[0117] In Figure 11, if the width of the region enclosed by the thick border is narrow (the distance between the horizontal division points is short), the amount of encoded data reduced in that region is not large. Nevertheless, if affine transformations and padding are applied to each of the nine subregions, the overall processing efficiency may decrease.

[0118] Therefore, in this embodiment, for regions with a narrow width, individual affine transformation and padding processes are not executed, as will be described later.

[0119] In the example shown in Figure 11, the focus is on the distance between division points in the horizontal direction. However, if the focus is on the distance between division points in the vertical direction, the same problem as described above with reference to Figure 11 arises for regions with limited height. Therefore, in this embodiment, the video encoding device prevents processing (such as affine transformation) from being performed on individual division regions, even for regions with limited height (for example, in Figure 11, regions that extend horizontally and have limited height).

[0120] Figures 12A and 12B are explanatory diagrams for illustrating the RTG processing of this embodiment. The content of Figure 12A is the same as that of Figure 11. However, in Figure 12A, straight lines extending from the division points that can be removed are shown as dashed lines. Figure 12B shows the subregions after the division points have been removed. In other words, the division point removal process is performed. To put it another way, in the horizontal direction (horizontal direction, i.e., x direction), the i-th division point is removed when the distance between the i-th division point and the (i+1)-th division point is short. Also, in the vertical direction (vertical direction, i.e., y direction), the j-th division point is removed when the distance between the j-th division point and the (j+1)-th division point is short. The number of subregions exemplified in Figure 12B is reduced compared to the number of subregions exemplified in Figure 11. Therefore, in this embodiment, the processing efficiency of the video encoder 100A is improved. Furthermore, the division point removal process can also be described as a merge process that combines each sub-region in a column with a narrow width (corresponding to each sub-region within the thick frame in Figure 11) with an adjacent sub-region.

[0121] Figure 13 is a block diagram showing an example of a video encoder 100A and a video decoder 200A suitable for a video encoding scheme for machines. Specifically, it is a block diagram showing an example of the configuration of a video encoder 100A that performs pre-processing and a video decoder 200A that performs post-processing.

[0122] The video encoder 100A comprises a time resampling unit 101, an RTG processing unit 102A, a spatial resampling unit 103, a bit depth shifting unit 104, an internal encoder 105, and a multiplexer 106. The configuration and operation of the time resampling unit 101, the spatial resampling unit 103, the bit depth shifting unit 104, the internal encoder 105, and the multiplexer 106 are the same as those of the video encoder 100 in the first embodiment shown in Figure 1. The operation of the RTG processing unit 102A differs from the operation of the RTG processing unit 102 in the first embodiment.

[0123] The video decoder 200A comprises a demultiplexer 201, an internal decoder 202, a spatial resampling unit 203, an RTG processing unit 204A, a temporal resampling unit 205, and a bit depth shifting unit 206. The configuration and operation of the demultiplexer 201, the internal decoder 202, the spatial resampling unit 203, the temporal resampling unit 205, and the bit depth shifting unit 206 are the same as those of the video decoder 200 in the first embodiment shown in Figure 1. The operation of the RTG processing unit 204A differs from the operation of the RTG processing unit 204 in the first embodiment.

[0124] Figure 14 is a block diagram showing an example configuration of the RTG processing unit 102A in the video encoder 100A. In the example configuration shown in Figure 14, the RTG processing unit 102A includes an ROI information acquisition unit 1021, a partial region definition unit 1022A, a parameter determination unit 1023, and a conversion unit 1024. The configuration and operation of the ROI information acquisition unit 1021, the parameter determination unit 1023, and the conversion unit 1024 are the same as their configuration and operation in the first embodiment shown in Figure 6. The configuration and operation of the partial region definition unit 1022A are different from the configuration and operation of the partial region definition unit 1022 in the first embodiment.

[0125] In this embodiment, as described above, the partial region definition unit 1022A removes one of the division points if the distance between the horizontal division points is short. Similarly, the partial region definition unit 1022A removes one of the division points if the distance between the vertical division points is short. A short distance between division points means, for example, that the distance between the division points is shorter than a predetermined value, i.e., below a predetermined threshold. The predetermined value can be set arbitrarily, but for example, a fixed value, or a value derived from the size of the picture or parameters in other processing (processing of the internal codec 105 or processing of the spatial resampling unit 103) can be adopted as the predetermined value. The predetermined value is supplied from the partial region definition unit 1022A to the multiplexer 106. The multiplexer 106 includes information that can identify the predetermined value in the side information transmitted to the video decoder 200A. Therefore, a bitstream containing information that can identify the predetermined value is transmitted to the video decoder 200A. Note that the horizontal threshold and the vertical threshold are the same value, but they may be different values.

[0126] Figure 15 is a block diagram showing an example configuration and operation of the RTG processing unit 204 in the video decoder 200A. In the example configuration shown in Figure 15, the RTG processing unit 204 includes an ROI information acquisition unit 2041, a parameter acquisition unit 2043, and a conversion unit 2044. The configuration and operation of the ROI information acquisition unit 2041, the parameter acquisition unit 2043, and the conversion unit 2044 are the same as their configuration and operation in the first embodiment shown in Figure 7. The operation of the partial region restoration unit 2042A is different from the operation of the partial region restoration unit 2042 in the first embodiment.

[0127] The partial region reconstruction unit 2042A, similar to the partial region reconstruction unit 2042 in the first embodiment, obtains division points from the size of the picture after RTG processing in the video encoder 100 and the ROI position information. Referring to the example shown in Figure 7, the partial region reconstruction unit 2042A reconstructs the division points shown in example (A). That is, the partial region reconstruction unit 2042 obtains multiple partial regions defined by multiple division points in the horizontal direction and multiple division points in the vertical direction, and defined by straight lines extending from the division points.

[0128] In the first embodiment, the partial region restoration unit 2042 reduced the division points using a process similar to that performed by the partial region definition unit 1022 in the video encoder 100. However, in this embodiment, the partial region restoration unit 2042A reduces the division points using a process similar to that performed by the partial region definition unit 1022A in the video encoder 100A.

[0129] In the video encoder 100A, when the sub-region definition unit 1022A performs a division point removal process as described with reference to Figures 12A and 12B, a bitstream containing encoded data in which affine transformation and encoding processing have been performed on a single sub-region obtained by combining multiple sub-regions, and a bitstream containing side information that allows the division points to be restored, are transmitted to the video decoder 200A. Therefore, the video decoder 200A can process multiple sub-regions all at once, similar to the video encoder 100A. As a result, the video decoder 200A can suppress a decrease in processing efficiency.

[0130] In the second embodiment, the side information capable of restoring the division points includes information capable of identifying a predetermined value (threshold) relating to the distance between the division points. Based on this predetermined value, the partial region restoration unit 2042A can remove the division points, similar to the partial region definition unit 1022A in the video encoder 100A.

[0131] Figure 16 is a flowchart showing the operation of the RTG processing unit 102A in the video encoder 100A.

[0132] The ROI information acquisition unit 1021 acquires ROI location information based on information that can identify the ROI, for example, included in the control information input to the video encoder 100 from an external source, as in the first embodiment (step S101). The partial region definition unit 1022A defines a partial region and determines the magnification of the partial region by performing the same processing as the partial region definition unit 1022 in the first embodiment (step S102).

[0133] Furthermore, the partial region definition unit 1022A performs a partial region merging process (the division point removal process illustrated in Figures 12A and 12B) as described above (step S103A).

[0134] The parameter determination unit 1023 determines the parameters for the affine transformation for each sub-region, similar to the first embodiment (step S104). The transformation unit 1024 performs the affine transformation operation for each sub-region based on the parameters for the affine transformation, similar to the first embodiment (step S105).

[0135] Figure 17 is a flowchart showing the operation of the RTG processing unit 204A in the video decoder 200A.

[0136] The ROI information acquisition unit 2041 acquires ROI information from the side information, similar to the first embodiment (step S201). The ROI information includes at least the position information of the partial region and the reduction ratio (magnification) of the ROI.

[0137] The partial region restoration unit 2042A, similar to the partial region restoration unit 2042 in the first embodiment, restores the position of the partial region based on the ROI position information, the ROI scaling factor, and information representing the relationship (difference) between the coordinates of the partial region after affine transformation (see Example (B) in Figure 7) and the original coordinates (see Example (A) in Figure 7), which are included in the side information (step S202A). Specifically, the partial region restoration unit 2042A restores the division points as illustrated in Example (A) in Figure 7.

[0138] In step S202A, the partial region restoration unit 2042A further reduces the division points using a process similar to that performed by the partial region definition unit 1022A in the video encoder 100A (see Figures 12A and 12B).

[0139] The parameter acquisition unit 2043 determines the parameters of the affine transformation for each subregion based on the side information, as in the first embodiment (step S203). The parameters to be determined are the amount of displacement (translation) and the magnification factor. The magnification factor is determined based on the magnification factor included in the side information.

[0140] The padding portion is subjected to an affine transformation for each sub-region, similar to the case of the first embodiment (step S204).

[0141] [Modification 1] In the video encoder 100A and video decoder 200A, when the partial region definition unit 1022A and the partial region restoration unit 2042A perform a merging process, the partial region containing the ROI and a specific region (partial region C: see Figure 6) may be merged to form a single partial region.

[0142] Figure 18 is an explanatory diagram illustrating a modified version of the RTG processing. In the example shown in Figure 18, the subregion containing ROI2 and the specific region are combined by the RTG processing (particularly the division point removal processing as a merging process as described above). That is, the division points between the subregion containing ROI2 and the specific region are removed by the merging process. When the decoding process is performed in this state, there is a possibility that pixels outside the subregion containing ROI2, i.e., pixels of the specific region, will be referenced. In that case, referencing pixels of a region with different properties, such as the specific region, may degrade the image quality of the subregion containing ROI2.

[0143] Therefore, in this modified example, the partial region restoration unit 2042A separates a single partial region formed by combining the partial region containing ROI2 and the specific region into the partial region containing ROI2 and the specific region. For example, the partial region restoration unit 2042A identifies the boundary between ROI2 and the specific region based on the ROI's position information. Then, it divides the partial region into two regions at that boundary. In other words, it prevents local combining operations from being performed based on the ROI's position information. As a result, cross-boundary references can be prevented during the decoding process for ROI2. In this case, since it simply prevents local combining operations from being performed, the number of partial regions does not increase compared to the number of partial regions before the combining operation.

[0144] In this example, the region was divided to a specific area, but similarly, when referencing pixels from regions with different properties, the merging process may be omitted. For example, if two or more ROIs with significantly different pixel values ​​overlap, the merging process may not be applied to the boundary area, even if the regions have the same magnification.

[0145] [Modification 2] In the second embodiment, the threshold used by the video encoder 100A to remove the split points is included in the bitstream as side information, and the video decoder 200A removes the split points in the same procedure as the video encoder 100A according to that threshold. If the method for determining this threshold is predetermined by the video encoder 100A and the video decoder 200A, the video encoder 100A may generate a bitstream that does not include the threshold in the side information, and the video encoder 100A and the video decoder 200A may perform the process of removing the split points using a value determined by the predetermined method as the threshold. For example, as described above, this is applicable when the threshold is a fixed value or a value derived from the size of the picture or parameters in other processing.

[0146] [Modification 3] In the second embodiment, the threshold itself was included in the bitstream as side information, but instead, parameters for deriving the threshold may be included in the bitstream. For example, the video encoder 100A includes the maximum number of division points in the horizontal and vertical directions in the bitstream, and the video encoder 100A and video decoder 200A determine a threshold in each direction such that the number of division points is less than or equal to the maximum number. As an example of this method, when there are K division points in a certain direction and the maximum number is T (K > T), the distances between division points are arranged in ascending order, and the distance between the K-T shortest division points is used as the threshold.

[0147] While each of the above embodiments can be implemented using hardware, it can also be realized using computer programs.

[0148] The information processing system shown in Figure 19 comprises a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. Multiple processors can also be used. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage media.

[0149] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiment.

[0150] The processor 701 then executes processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoders 100, 100A or the video decoders 200, 200A as shown in the above embodiment.

[0151] The processor 701 performs processing according to a video encoding program that realizes the functions of each block in the video encoder 100 shown in Figure 1, thereby realizing the functions of the video encoder 100. In addition, the processor 701 can realize the functions of the video encoder 100, including the RTG processing unit 102 shown in Figure 6, by performing processing according to the program.

[0152] Furthermore, the functions of the video encoder 100A are realized when the processor 701 executes processing according to a video encoding program that realizes the functions of each block in the video encoder 100A shown in Figure 13. In addition, the processor 701 can realize the functions of the video encoder 100, including the RTG processing unit 102A shown in Figure 14, by executing processing according to the program.

[0153] Furthermore, the functions of the video decoder 200 are realized when the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoder 200 shown in Figure 1. In addition, the processor 701 can realize the functions of the video decoder 200, including the RTG processing unit 204 shown in Figure 7, by executing processing according to the program.

[0154] Furthermore, the functions of the video decoder 200A are realized when the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoder 200A shown in Figure 13. In addition, the processor 701 can realize the functions of the video decoder 200, including the RTG processing unit 204A shown in Figure 15, by executing processing according to the program.

[0155] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0156] Figure 20 is a block diagram showing the main part of the video encoding device. The video encoding device 10 shown in Figure 20 (implemented by a video encoder 100A in this embodiment) includes a sub-region definition means 11 (implemented by a sub-region definition unit 1022A in this embodiment) which defines a plurality of sub-regions defined by straight lines extending from a division point in a picture to be processed, defining a plurality of division points in the horizontal direction and setting a plurality of division points in the vertical direction; a division point removal means 12 (implemented by a sub-region definition unit 1022A in this embodiment) which combines horizontally adjacent sub-regions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combines vertically adjacent sub-regions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than a predetermined value; and a conversion means 13 (implemented by a conversion unit 1024 in this embodiment) which performs an affine transformation operation for each sub-region.

[0157] Figure 21 is a block diagram showing the main part of the video decoding device. The video decoding device 20 shown in Figure 21 (implemented by the video decoder 200A) comprises a decoding means 21 (implemented by the internal decoder 202 in this embodiment) that decodes a video bitstream and acquires a decoded picture; an acquisition means 22 (implemented by the partial region restoration unit 2042A in this embodiment) that acquires from the decoded picture a plurality of subregions defined from the decoded picture, where a plurality of division points are defined in the horizontal direction and a plurality of division points are defined in the vertical direction and defined by straight lines extending from the division points; a division point removal means 23 (implemented by the partial region restoration unit 2042A in this embodiment) that combines horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combines vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than a predetermined value; and a conversion means 24 (implemented by the conversion unit 2044 in this embodiment) that performs an affine transformation operation for each subregion.

[0158] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0159] (Note 1) A video encoding method comprising defining multiple division points in the horizontal direction and setting multiple division points in the vertical direction in a picture to be processed, defining multiple subregions defined by straight lines extending from the division points, combining horizontally adjacent subregions by removing one of the division points when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, combining vertically adjacent subregions by removing one of the division points when the distance between adjacent division points in the vertical direction is shorter than the predetermined value, and performing an affine transformation operation for each of the subregions.

[0160] (Note 2) The video encoding method according to Note 1, which transmits a bitstream containing information that can identify the predetermined value.

[0161] (Note 3) The video encoding method according to Note 1 or Note 2, wherein a division point is set based on the region of interest in the picture to be processed, and when a subregion that does not include the region of interest and a subregion that includes the region of interest are combined into one subregion, the one subregion is separated into a subregion that does not include the region of interest and a subregion that includes the region of interest.

[0162] (Note 4) A video decoding method comprising: decoding a video bitstream to obtain a decoded picture; obtaining a plurality of subregions from the decoded picture in which a plurality of division points are defined in the horizontal direction and a plurality of division points are defined in the vertical direction and defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one of the division points when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value; combining vertically adjacent subregions by removing one of the division points when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and performing an affine transform operation for each of the subregions.

[0163] (Note 5) The video decoding method described in Note 4, which extracts information from the bitstream that allows for the identification of the predetermined value.

[0164] (Note 6) The video decoding method according to Note 4 or Note 5, which, when acquiring a decoded picture in which division points are set based on a region of interest, separates a single subregion into a subregion that does not include the region of interest and a subregion that includes the region of interest when the subregion that does not include the region of interest and the subregion that includes the region of interest are combined into a single subregion.

[0165] (Note 7) A video encoding device comprising: sub-region definition means for defining multiple division points in the horizontal direction and setting multiple division points in the vertical direction in a picture to be processed, and defining multiple sub-regions defined by straight lines extending from the division points; division point removal means for combining horizontally adjacent sub-regions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent sub-regions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and transformation means for performing an affine transformation operation for each of the sub-regions.

[0166] (Note 8) The video encoding apparatus according to Note 7, comprising bitstream creation means (in this embodiment, a partial region definition unit 1022 and a multiplexer 106) for creating a bitstream containing information that can identify the predetermined value.

[0167] (Note 9) The video encoding apparatus according to Note 7 or Note 8, wherein the subregion definition means sets division points based on the region of interest in the picture to be processed, and when a subregion that does not include the region of interest and a subregion that includes the region of interest are combined into one subregion, the single subregion is separated into a subregion that does not include the region of interest and a subregion that includes the region of interest (implemented in the embodiment by the subregion definition unit 1022A).

[0168] (Note 10) A video decoding device comprising: a decoding means for decoding a video bitstream and obtaining a decoded picture; an acquisition means for obtaining a plurality of subregions from the decoded picture in which a plurality of division points are defined in the horizontal direction and a plurality of division points are defined in the vertical direction and defined by straight lines extending from the division points; a division point removal means for combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and a conversion means for performing an affine transformation operation for each of the subregions.

[0169] (Note 11) The video decoding device according to Note 10, further comprising extraction means for extracting information that can identify the predetermined value from the bitstream (implemented in the embodiment by a demultiplexer 201).

[0170] (Note 12) When acquiring a decoded picture in which division points are set based on a region of interest, the video decoding device according to Note 10 or Note 11 is provided with separation means (implemented in the embodiment by a partial region restoration unit 2042A) for separating a single partial region into a partial region that does not include the region of interest and a partial region that includes the region of interest when the single partial region is combined with a partial region that does not include the region of interest.

[0171] (Note 13) A video encoding program for a computer to perform the following processes: defining multiple division points in the horizontal direction and setting multiple division points in the vertical direction in a picture to be processed, and defining multiple subregions defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one of the division points when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent subregions by removing one of the division points when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and performing an affine transformation operation for each of the subregions.

[0172] (Note 14) A video decoding program for a computer to perform the following processes: decoding a video bitstream and obtaining a decoded picture; obtaining from the decoded picture a plurality of subregions defined by a plurality of division points in the horizontal direction and a plurality of division points in the vertical direction, and defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and performing an affine transformation operation for each of the subregions.

[0173] Some or all of the configurations described in Appendices 2 and 3, which are dependent on Appendice 1 above, and Appendices 5 and 6, which are dependent on Appendice 4 above, can be applied to various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the embodiments described above.

[0174] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0175] This application claims priority based on Japanese Patent Application No. 2025-003952, filed on 10 January 2025, and incorporates all of its disclosures herein.

[0176] 10 Video encoding device 11 Partial region definition means 12 Division point removal means 13 Conversion means 20 Video decoding device 21 Decoding means 22 Acquisition means 23 Division point removal means 24 Conversion means 100, 100A Video encoder 101 Time resampling unit 102, 102A RTG processing unit 103 Spatial resampling unit 104 Bit depth shift unit 105 Internal encoder 106 Multiplexer 200, 200A Video decoder 201 Demultiplexer 202 Internal decoder 203 Spatial resampling unit 204, 204A RTG processing unit 205 Time resampling unit 206 Bit depth shift unit 701 Processor 702 Program memory 703, 704 Storage medium 1021 ROI information acquisition unit 1022, 1022A Partial region definition unit 1023 Parameter determination unit 1024 Conversion unit 2041 ROI information acquisition unit 2042, 2042A Partial region restoration unit 2043 Parameter acquisition unit 2044 Conversion unit

Claims

1. A video encoding method comprising: defining multiple division points in the horizontal direction and setting multiple division points in the vertical direction in a picture to be processed; defining multiple subregions defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value; combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and performing an affine transformation operation for each of the subregions.

2. The video encoding method according to claim 1, which transmits a bitstream containing information that can identify the predetermined value.

3. The video encoding method according to claim 1 or 2, wherein a division point is set based on the region of interest in the picture to be processed, and when a subregion that does not include the region of interest and a subregion that includes the region of interest are combined into a single subregion, the single subregion is separated into a subregion that does not include the region of interest and a subregion that includes the region of interest.

4. A video decoding method comprising: decoding a video bitstream to obtain a decoded picture; obtaining multiple subregions from the decoded picture, each subregion defined by multiple division points in the horizontal direction and multiple division points in the vertical direction, and defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value; combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and performing an affine transform operation for each subregion.

5. The video decoding method according to claim 4, wherein information that can identify the predetermined value is extracted from the bitstream.

6. The video decoding method according to claim 4 or 5, wherein when a decoded picture is obtained in which division points are set based on a region of interest, and a subregion that does not include the region of interest and a subregion that includes the region of interest are combined into a single subregion, the single subregion is separated into a subregion that does not include the region of interest and a subregion that includes the region of interest.

7. A video encoding device comprising: sub-region definition means for defining multiple division points in the horizontal direction and setting multiple division points in the vertical direction in a picture to be processed, and defining multiple sub-regions defined by straight lines extending from the division points; division point removal means for combining horizontally adjacent sub-regions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent sub-regions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and transformation means for performing an affine transformation operation for each of the sub-regions.

8. The video encoding apparatus according to claim 7, further comprising bitstream creation means for creating a bitstream containing information that can identify the predetermined value.

9. The video encoding apparatus according to claim 7 or 8, wherein the subregion definition means sets division points based on the region of interest in the picture to be processed, and when a subregion that does not include the region of interest and a subregion that includes the region of interest are combined into a single subregion, the single subregion is separated into a subregion that does not include the region of interest and a subregion that includes the region of interest.

10. A video decoding device comprising: a decoding means for decoding a video bitstream and obtaining a decoded picture; an acquisition means for obtaining a plurality of subregions from the decoded picture, each subregion defined by a plurality of division points in the horizontal direction and a plurality of division points in the vertical direction, and defined by straight lines extending from the division points; a division point removal means for combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and a conversion means for performing an affine transformation operation for each subregion.

11. The video decoding apparatus according to claim 10, further comprising extraction means for extracting information from a bitstream that can identify the predetermined value.

12. When acquiring a decoded picture in which division points are set based on a region of interest, the video decoding device according to claim 10 or claim 11 is further provided with separation means for separating a single subregion into a subregion that does not include the region of interest and a subregion that includes the region of interest, when the subregion that does not include the region of interest and a subregion that includes the region of interest are combined into a single subregion.

13. A video encoding program for a computer to perform the following processes: defining multiple division points in the horizontal direction and setting multiple division points in the vertical direction in a picture to be processed, and defining multiple subregions defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one division point when the distance between adjacent division points in the horizontal direction is shorter than a predetermined value, and combining vertically adjacent subregions by removing one division point when the distance between adjacent division points in the vertical direction is shorter than the predetermined value; and performing an affine transformation operation for each of the subregions.

14. A video decoding program for a computer to perform the following processes: decoding a video bitstream and obtaining a decoded picture; obtaining from the decoded picture multiple subregions defined by multiple division points in the horizontal direction and multiple division points in the vertical direction, and defined by straight lines extending from the division points; combining horizontally adjacent subregions by removing one division point when the distance between adjacent horizontal division points is shorter than a predetermined value, and combining vertically adjacent subregions by removing one division point when the distance between adjacent vertical division points is shorter than the predetermined value; and performing an affine transformation operation for each of the subregions.

Images