Image coding device and image coding method

Abstract

This image coding device determines whether or not an object region detected from an image is a region of interest; determines a coding parameter for controlling the image quality of a coded image; corrects the coding parameter in accordance with the degree of overlap between a CTU, which is a divided image of the image, and the region of interest; determines a coding mode and a division shape of the CTU, the coding mode being used for generating a predicted image of the image from a quantized image, so that the value of a cost function calculated along with coding according to the corrected coding parameter is minimized; and codes the image for each CTU using the division shape of the CTU and the coding mode that have been determined.

Description

Image encoding device and image encoding method

[0001] The disclosed technology relates to an image encoding device and an image encoding method.

[0002] The need for video analysis using AI (Artificial Intelligence) is increasing, and AI is expected to be applied to various fields such as autonomous driving and surveillance monitoring of vehicles.

[0003] Generally, advanced video analysis using AI tends to have a large amount of computation. Therefore, a system that realizes advanced processing through the cooperation of a cloud with few resource constraints and excellent scalability and an edge terminal with excellent real-time performance has attracted attention.

[0004] For example, an event-driven inference system has been proposed in which an event is detected by performing AI inference processing with relatively little computation on an edge terminal, the video is transmitted to the cloud when the edge terminal detects the event, and more advanced AI inference processing is performed in the cloud. As a result, unnecessary data transmission between the edge terminal and the cloud and the resource consumption in the cloud are reduced.

[0005] On the other hand, even in the above-described event-driven inference system, when events are frequently detected by a large number of edge terminals, the transmission cost when transmitting the video to the cloud can be a problem. Therefore, it is desirable for the edge terminal to compress and transmit the video as much as possible while considering a decrease in the inference accuracy in the cloud.

[0006] As a method for reducing the video transmission bandwidth while maintaining the inference accuracy on the cloud side as much as possible, for example, ROI (Region of Interest) encoding technology has been proposed (see, for example, Non-Patent Document 1).

[0007] In the technology described in Non-Patent Literature 1, an area important for AI processing is detected using an object detector such as YOLO (You Only Look Once), and the CTU (Coding Tree Unit) that overlaps with the obtained object detection frame (hereinafter referred to as Bbox (BoundingBox)) is used to determine whether or not a CTU is an ROI. In the technology described in Non-Patent Literature 1, a CTU that is not an ROI, i.e., a QP that is uniformly lower than the quantization parameter (QP) applied to non-ROIs, is applied to the identified ROI. In other words, the technology described in Non-Patent Literature 1 reduces the video transmission bitrate while maintaining the AI ​​inference accuracy for the encoded video by lowering the compression ratio only in the image area important for AI processing to improve image quality, and increasing the compression ratio in other areas to lower image quality.

[0008] Here, CTU refers to the basic unit of encoding, which is a spatially equal division of the video frame to be encoded. For example, the maximum CTU size in H.265 / HEVC (High Efficiency Video Coding) is 64x64. In H.266 / VVC (Versatile Video Coding), the maximum CTU size is 128x128. Generally, the larger the CTU size, the higher the encoding efficiency, which is advantageous in terms of image quality and compression ratio.

[0009] In the aforementioned video encoding standards, the CTU is further divided into units called CUs (Coding Units). In HEVC, each CTU is divided based on a quad-tree block partition (hereafter referred to as a QT (Quad Tree) partition). Therefore, the possible sizes of CUs are 8x8, 16x16, 32x32, and 64x64, and a CTU is constructed by a combination of CUs of each size.

[0010] On the other hand, in VVC, each CTU can be divided not only into quadtree block partitions, but also into binary tree block partitions (hereinafter referred to as BT (Binary Tree) partitions) which divide the block into two horizontally or vertically, and ternary tree block partitions (hereinafter referred to as TT (Triple Tree) partitions) which divide the block into three horizontally or vertically with an edge ratio of 1:2:1. Therefore, the size of the CU in VVC can be selected from a wider variety of sizes than in HEVC, such as 32x4 or 8x16, and in VCC, the CTU is composed of combinations of these CU sizes.

[0011] Furthermore, HEVC and VVC reference software employ Rate-Distortion (RD) optimization in determining the division shape of CUs within the CTU and the encoding mode for each CU. RD optimization is a method of selecting the division shape and encoding mode of a CU so as to minimize the cost value (hereinafter referred to as RD cost) obtained by performing processes such as prediction, transformation, and quantization on the CU, and combining the resulting code amount with the image distortion associated with encoding. For example, in VVC reference software, when determining the size of each CU within the CTU, prediction, transformation, and quantization are performed on a total of six division shapes: no division, QT division, BT division (vertical and horizontal), and TT division (vertical and horizontal) using different encoding modes such as intra-prediction and inter-prediction, to determine the division shape and encoding mode that minimize the RD cost.

[0012] Furthermore, the technology described in Non-Patent Document 2 identifies ROI and non-ROI using CU as the unit and controls QP, thereby controlling the compression ratio and image quality in spatially finer image area units than the technology described in Non-Patent Document 1, and aiming to reduce the video transmission bitrate.

[0013] <Non-patent document 1> Kristian Fischer, etc. “Saliency-Driven Versatile Video Coding for Neural Object Detection,” 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021 <Non-patent document 2> Wei Li, etc. “Saliency-Enabled Coding Unit Partitioning and Quantization Control for Versatile Video Coding,” Information 2022

[0014] In the technology described in Non-Patent Document 1, the compression ratio and image quality are controlled for each CTU, but compared to the technology described in Non-Patent Document 2, where the CU is used as the unit for controlling the compression ratio and image quality, the control is performed on a spatially coarser image region basis. Therefore, it is not possible to improve image quality by focusing on an area smaller than the size of the CTU, and the video transmission bitrate may increase unnecessarily.

[0015] On the other hand, while the technology described in Non-Patent Literature 2 enables control at a finer image region unit than CTU, RD optimization is not used in most cases when determining the division shape and encoding mode when dividing the CTU into CUs. Therefore, the technology described in Non-Patent Literature 2 may result in lower encoding efficiency, image quality degradation, and an increase in the video transmission bitrate compared to when encoding is performed using the division shape and encoding mode of CUs obtained by RD optimization.

[0016] The disclosed technology was developed in view of the above points, and aims to provide an image encoding device and image encoding method that can reduce video transmission bandwidth while maintaining as much as possible the inference accuracy on the cloud side for decoded video after video encoding, compared to conventional event-driven inference systems.

[0017] A first aspect of this disclosure is an image encoding apparatus comprising: a determination unit that determines whether an object region detected from an image is a region of interest that includes a target for processing; a parameter determination unit that determines encoding parameters for controlling the image quality after encoding; a modification unit that modifies the encoding parameters determined by the parameter determination unit according to the degree of overlap between the CTU, which is a divided image of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule; and an encoding unit that determines an encoding mode, which is a group of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTU, so as to minimize the value of a predetermined cost function calculated in conjunction with encoding according to the encoding parameters modified by the modification unit, and encodes the image for each CTU using the determined division shape of the CTU and the encoding mode.

[0018] A second aspect of the present disclosure is an image coding method, wherein a computer performs the following steps: determine whether an object region detected in an image is a region of interest that includes a target for processing; determine coding parameters to control the image quality after coding; modify the determined coding parameters according to the degree of overlap between the CTUs, which are the divided images of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the determined region of interest; determine a coding mode, which is a set of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTUs, such that the value of a predetermined cost function calculated in connection with coding according to the modified coding parameters is minimized; and encode the image for each CTU using the determined division shape of the CTUs and the coding mode.

[0019] According to the disclosed technology, compared to conventional event-driven inference systems, it is possible to reduce video transmission bandwidth while maintaining as much as possible the inference accuracy on the cloud side for decoded video after video encoding.

[0020] This is a diagram showing an example configuration of an event-driven inference system. This is a diagram showing an example configuration of an edge terminal. This is a diagram showing an example of the functional configuration of an image encoding device. This is a diagram showing a detailed functional configuration example of the encoding unit. This is a diagram showing an example of CTU division by VVC. This is a diagram showing an example of division where the CTU is recursively divided based on six types of division patterns. This is a diagram showing an example of the CTU division result. This is a diagram showing a detailed functional configuration example of the control unit. This is a diagram showing an example of the positional relationship between Bbox, the region of interest, and the CTU. This is a diagram showing an example of the main components of the electrical system of the image encoding device. This is a flowchart showing an example of the image encoding process flow. This is a flowchart showing an example of the encoding process flow. This is a flowchart showing an example of the correction process flow. This is a diagram showing an example of how to determine the division shape and encoding mode of the CTU when the overlap rate between the CTU and the region of interest is below a threshold. This is a diagram showing a modified version of the method for determining the division shape and encoding mode of the CTU when the overlap rate between the CTU and the region of interest exceeds a threshold. This is a diagram showing an example of how to determine the division shape and encoding mode of the CTU when the overlap rate between the CTU and the region of interest is below a threshold. This figure shows a modified example of a method for determining the partition shape and encoding mode of a CTU when the overlap rate between the CTU and the region of interest exceeds a threshold.

[0021] This embodiment will be described below with reference to the drawings. The same reference numerals are used throughout the drawings for the same components and processes, and redundant explanations are omitted. The dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0022] Figure 1 shows an example configuration of an event-driven inference system 1. As shown in Figure 1, the event-driven inference system 1 comprises a cloud 3 and at least one edge terminal 4. The type of edge terminal 4 used is not restricted, as it is a terminal suitable for the content of the events to be monitored and the content to be inferred. There is also no restriction on the number of edge terminals 4 included in the event-driven inference system 1. As an example, the edge terminal 4 includes equipment that captures images, such as surveillance cameras, drones, automobiles, and mobile terminals. In this embodiment, when describing surveillance cameras, drones, automobiles, and mobile terminals without distinction, the general term "edge terminal 4" is used.

[0023] In the following section, the event-driven inference system 1 will be explained using an example where the edge terminal 4 performs object detection, which is one of the predetermined processes performed on video.

[0024] The edge terminal 4 is connected to the cloud 3 wirelessly via a network (not shown). The edge terminal 4 detects predetermined events by performing predetermined processing and transmits the video captured by the camera to the cloud 3 at the time the event is detected. In this embodiment, "video" refers to a moving image containing multiple images. Depending on the situation, an image may be referred to as a "frame".

[0025] The cloud 3, which receives video from the edge terminal 4, is what is known as cloud computing and is built in a data center or similar location.

[0026] Cloud 3 performs predetermined inference processing on the video received from the edge terminal 4. The inference results obtained from Cloud 3's inference processing are used by the collaborative server 2, which runs other services. Therefore, Cloud 3 transmits the inference results to the collaborative server 2 via the network. Cloud 3 can also control the operation of the edge terminal 4 by sending instructions to the edge terminal 4 according to the obtained inference results.

[0027] Figure 2 shows an example configuration of an edge terminal 4 according to this embodiment. The edge terminal 4 includes, for example, a camera 5, an image encoding device 6, a communication unit 7, and storage 8.

[0028] Camera 5 is an example of a shooting device that captures images at the edge terminal 4. Camera 5 outputs the images obtained from the capture to the image encoding device 6. In this embodiment, the image encoding device 6 processes the images captured by Camera 5, but the images processed by the image encoding device 6 are not limited to images captured by Camera 5. For example, moving images such as animations composed of multiple images drawn by a user at the edge terminal 4 may be processed by the image encoding device 6. The images input from Camera 5 to the image encoding device 6 are an example of moving images according to this disclosure.

[0029] The image encoding device 6 is a device that performs image encoding and predetermined processing on the video input from the camera 5, and is, for example, an accelerator. The image encoding device 6 outputs the detection results obtained from object detection from the images that make up the video, and the bitstream obtained from image encoding, to the storage 8.

[0030] The storage device 8 stores the detection results and bitstream obtained by the image encoding device 6. The storage device 8 also stores the image encoding program executed by the image encoding device 6. The image encoding program may be a single program or a group of programs consisting of multiple programs or modules. For example, the storage device 8 may be a DRAM (Dynamic Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive). The image encoding program of this embodiment is an example of the program of this disclosure.

[0031] Based on instructions from the control unit 13 (see Figure 3) provided in the image encoding device 6, the communication unit 7 reads the bitstream stored in the storage 8 and transmits it to the cloud 3 at the time it detects an event.

[0032] Next, the functional configuration of the image encoding device 6 will be described. Figure 3 is a diagram showing an example of the functional configuration of the image encoding device 6. As shown in Figure 3, the image encoding device 6 comprises the functional units of a preprocessing unit 10, an encoding unit 11, a detection unit 12, and a control unit 13.

[0033] The preprocessor 10 converts the video input from the camera 5 into a format that the encoding unit 11 can process, and outputs the converted video to the storage 8 via the memory interface 6G (see Figure 10). The preprocessor 10 also converts the video input from the camera 5 into a format that the detection unit 12 can process, and outputs the converted video to the storage 8 via the memory interface 6G.

[0034] For example, the preprocessing unit 10 converts the video input from the camera 5 into the YUV420 format, which can be processed by the encoding unit 11. The preprocessing unit 10 also converts the video input from the camera 5 into the RGB format, which can be processed by the detection unit 12.

[0035] For the sake of explanation, the image before encoding by the encoding unit 11 will be referred to as the "original image." In other words, the image that constitutes the video whose format has been converted by the preprocessing unit 10 is an example of an original image. Furthermore, from here on, the memory interface 6G will be referred to as memory I / F (Interface) 6G.

[0036] The encoding unit 11 encodes each region, which is an encoding block within a frame, using encoding parameters. Specifically, the encoding unit 11 compresses and encodes the video input from the camera 5 using a lossy encoding method. The encoding unit 11 is a so-called encoder. The encoding unit 11 reads the video stored in the storage 8 by the preprocessing unit 10, performs compression encoding in accordance with video encoding standards such as H.264 / AVC (Advanced Video Coding), H.265 / HEVC, and H.266 / VVC, and outputs the bitstream obtained by the compression encoding to the storage 8. The encoding unit 11 performs compression encoding of the image in units of regions, which are encoding blocks, according to the encoding parameters determined by the control unit 13, which will be described later. For example, CTU (Coding Tree Unit) 14 and CU (Coding Unit) 15 are used as regions for which compression encoding is performed (see Figure 7). CTU14 is an example of a divided image obtained after dividing an image into multiple blocks according to a predetermined division rule. CU15 is an example of a sub-region obtained by further dividing CTU14.

[0037] The detection unit 12 detects specific objects within each frame that makes up the video. Specifically, the detection unit 12 performs object detection processing to detect predetermined objects from the original image and outputs the detection results to the storage 8. The detection unit 12 is, for example, an AI inference core that performs inference using an object detection model pre-trained by a neural network. It detects the position and size of specific objects, such as a person or a car, that have been pre-specified by the user, from the original image and outputs them as Bbox 16 (see Figure 9).

[0038] The processing that can be performed by the detection unit 12 is not limited to object detection; for example, it can also perform tracking of specific objects, segmentation of each object, and generation of importance maps. If the detection unit 12 does not directly output the position or size of a specific object, such as when the detection result output by the detection unit 12 is an importance map, the position and size of the specific object may be determined by performing post-processing on the importance map. For example, the detection unit 12 may consider the position and number of pixels in a group of pixels where the importance of each pixel shown in the importance map is above a predetermined threshold as the position and size of the specific object.

[0039] The control unit 13 determines the encoding parameters for each region used by the encoding unit 11 in accordance with a predetermined target code amount. The control unit 13 uses quantization parameters as an example of encoding parameters. Therefore, hereafter, encoding parameters will be referred to as "encoding parameter QP (Quantize Parameter)". However, encoding parameter QP is not limited to quantization parameters. In general, there is a strong correlation between the target code amount and encoding parameter QP, and encoding is performed by setting the encoding parameter QP higher when the target code amount is small and lower when the target code amount is large. Therefore, the target code amount may be used directly as encoding parameter QP. The control unit 13 may dynamically change the encoding parameter QP for each region based on multiple detection results output by the detection unit 12.

[0040] The preprocessing unit 10, the encoding unit 11, and the detection unit 12 may each be implemented by a dedicated accelerator.

[0041] Next, the encoding unit 11 will be described in detail. Figure 4 is a diagram showing an example of the detailed functional configuration of the encoding unit 11. The encoding unit 11 includes a division shape / mode determination unit 11A, a residual signal generation unit 11B, a frequency conversion unit 11C, a quantization unit 11D, an entropy encoding unit 11E, an inverse quantization unit 11F, an inverse frequency conversion unit 11G, a prediction image generation unit 11H, a decoded image generation unit 11J, a loop filter processing unit 11K, and a reference image buffer unit 11L.

[0042] The split shape and mode determination unit 11A divides the original image whose format has been converted by the preprocessing unit 10 into CTUs 14 having a predetermined size according to the video coding standard, and determines the split shape and coding mode of the CUs 15 within the CTU 14.

[0043] In VVC, the original image is divided into CTUs 14 with a range of 128 pixels along the horizontal and vertical directions of the original image as one block. The CTU 14 is further block-divided into CUs 15 which are blocks obtained by recursively dividing the CTU 14, and a process of encoding with one CU 15 as a unit is performed.

[0044] Hereinafter, the sizes of the CTU 14 and the CU 15 are represented by "(number of pixels in the horizontal direction) × (number of pixels in the vertical direction)". For example, a CTU 14 with a range of 128 pixels along the horizontal and vertical directions as one block is represented as 128×128. In the case of VVC, since the CTU 14 is always QT-divided once, the maximum size of the CU 15 is 64×64. Also, in the case of VVC, not only CUs 15 with 64×64, 32×32, 16×16, and 8×8 obtained by recursively dividing the CTU 15 into four parts as one block, but also CUs 15 divided into two or three parts in the horizontal or vertical direction exist. FIG. 5 is a diagram showing an example of the division of the CTU 14 by VVC.

[0045] In FIG. 5, the CU 15A is a CU 15 that is not further divided, the CU 15B is a CU 15 obtained by dividing the original CU 15 into two parts in the horizontal direction. Also, the CU 15C is a CU 15 obtained by dividing the original CU 15 into two parts in the vertical direction, and the CU 15D is a CU 15 obtained by dividing the original CU 15 into four parts. Also, the CU 15E is a CU 15 obtained by dividing the original CU 15 into three parts in the horizontal direction with a side ratio of 1:2:1, and the CU 15F is a CU 15 obtained by dividing the original CU 15 into three parts in the vertical direction with a side ratio of 1:2:1.

[0046] FIG. 6 is a diagram showing an example of recursively dividing CTU14 based on the six types of division patterns shown in FIG. 5. The circles in FIG. 6 represent the nodes of the division tree. At each node in FIG. 6, one division pattern is selected from the six division patterns shown in FIG. 5. Each node in FIG. 6 is assigned reference numerals 15A to 15F, and the reference numeral of each node means performing the division shown in FIG. 5 in which the same reference numeral is assigned to the original CU15.

[0047] FIG. 7 is a diagram showing an example of a division result of dividing CTU14 according to the division example shown in FIG. 6. The numbers inside the circles in FIG. 6 correspond to the CUs 15 constituting the CTU14 shown in FIG. 7.

[0048] On the other hand, in HEVC, CTU14 is only recursively divided into four parts, so it can be seen that VVC realizes highly efficient encoding processing by having a more flexible block division pattern than HEVC.

[0049] The division shape / mode determination unit 11A acquires the original image to be encoded from the storage 8 and also acquires the decoded image from the reference image buffer unit 11L described later, and divides the CTU14 included in the original image into CUs 15. Further, the division shape / mode determination unit 11A outputs the original image divided into the sizes of the respective CUs 15 to the residual signal generation unit 11B, and outputs the size of the CU 15, that is, the division shape of the CU 15 and the encoding mode of the CU 15 to the predicted image generation unit 11H.

[0050] As an example of the encoding mode, there are encoding modes such as intra prediction that generates a predicted image by referring to decoded pixels within the same frame, and inter prediction that generates a predicted image by referring to different decoded frames. In order to generate a predicted image, more detailed parameters such as a prediction direction for determining the position and direction of pixels to be referred to in the case of intra prediction, and a motion vector for specifying the position of reference pixels within the frame to be referred to in the case of inter prediction need to be determined. In the present embodiment, a group of parameters necessary for generating a predicted image is collectively referred to as an "encoding mode".

[0051] The residual signal generation unit 11B acquires the original image divided into each CU 15 output from the division shape / mode determination unit 11A, and also acquires a predicted image corresponding to each CU 15 of the original image from the predicted image generation unit 11H. The residual signal generation unit 11B outputs the difference between the CU 15 in the acquired original image and the CU 15 in the predicted image corresponding to the CU 15 in the original image as a predicted residual signal to the frequency conversion unit 11C.

[0052] The frequency conversion unit 11C acquires the predicted residual signal from the residual signal generation unit 11B and performs a frequency conversion, for example, using a discrete cosine transform or discrete sine transform, for both the horizontal and vertical directions of the predicted image. The frequency conversion unit 11C outputs the group of conversion coefficients derived as a result of the frequency conversion of the predicted image to the quantization unit 11D.

[0053] The quantization unit 11D obtains a group of conversion coefficients from the frequency conversion unit 11C and quantizes the group of conversion coefficients using the coding parameter QP determined by the control unit 13. The quantization unit 11D outputs the quantized group of conversion coefficients to the entropy coding unit 11E and the inverse quantization unit 11F.

[0054] The entropy coding unit 11E obtains a group of conversion coefficients quantized by the quantization unit 11D, and performs entropy coding on the quantized group of conversion coefficients to derive a bitstream. The entropy coding unit 11E outputs the bitstream derived as a result of entropy coding to the storage 8.

[0055] The inverse quantization unit 11F obtains a group of quantized conversion coefficients from the quantization unit 11D, and performs an inverse quantization process on the quantized group of conversion coefficients to generate a group of conversion coefficients. The inverse quantization unit 11F outputs the group of conversion coefficients obtained as a result of the inverse quantization process to the inverse frequency conversion unit 11G.

[0056] The inverse frequency conversion unit 11G acquires a group of conversion coefficients from the inverse quantization unit 11F and performs an inverse frequency conversion on the group of conversion coefficients to generate a predicted residual signal. The inverse frequency conversion unit 11G outputs the predicted residual signal to the decoded image generation unit 11J.

[0057] The decoded image generation unit 11J acquires a predicted residual signal from the inverse frequency conversion unit 11G and obtains a predicted image of the original image from the predicted image generation unit 11H, which will be described later. The decoded image generation unit 11J generates a decoded image of the original image for each CU 15 by adding the predicted residual signal to the predicted image. The decoded image generation unit 11J outputs the decoded image of the original image to the loop filter processing unit 11K and the reference image buffer unit 11L.

[0058] The loop filter processing unit 11K acquires the decoded image from the decoded image generation unit 11J and performs a filter process on the decoded image to reduce image distortion caused by encoding. The loop filter processing unit 11K outputs the decoded image after the filter process to the reference image buffer unit 11L.

[0059] The reference image buffer unit 11L acquires the decoded image before filtering from the decoded image generation unit 11J and the decoded image after filtering from the loop filter processing unit 11K. The reference image buffer unit 11L stores the acquired decoded image before filtering and the decoded image after filtering as reference images. In other words, the reference image buffer unit 11L stores the decoded image obtained by decoding a past original image as a reference image.

[0060] The predictive image generation unit 11H obtains the encoding mode of the CU 15 related to the original image from the division shape / mode determination unit 11A, and also obtains past decoded images stored as reference images from the reference image buffer unit 11L. Based on the encoding mode of the original image and the past decoded images, the predictive image generation unit 11H generates a predicted image of the original image and outputs the generated predicted image to the residual signal generation unit 11B.

[0061] In other words, the encoding unit 11 determines the division shape and encoding mode of the CTU 14, and encodes the image for each CTU 14 using the determined division shape and encoding mode of the CTU 14.

[0062] Next, the control unit 13 shown in Figure 3 will be described in detail. Figure 8 is a diagram showing an example of the detailed functional configuration of the control unit 13. The control unit 13 includes a determination unit 13A, a parameter determination unit 13B, a calculation unit 13C, and a correction unit 13D.

[0063] The determination unit 13A uses the detection results of the detection unit 12, which detects specific objects within each frame constituting the video, to determine whether the object region detected from each frame is a region of interest 17A (see Figure 9) that includes the object to be processed. Specifically, the determination unit 13A refers to the detection results output from the detection unit 12 and determines that the image region surrounded by Bbox 16 and the image region adjacent to Bbox 16 obtained as detection results are regions of interest 17A, and determines that the other regions are regions of non-interest 17B (see Figure 9). The image region adjacent to Bbox 16 refers to the region outside of Bbox 16 that is included in the range that is a specified number of pixels away from the boundary line of Bbox 16. The specified number of pixels is, for example, a value that is set in advance by the user and stored in storage 8. If the specified number of pixels is set to "0", the image region surrounded by Bbox 16 itself becomes the region of interest 17A.

[0064] The parameter determination unit 13B determines a reference encoding parameter QP for controlling the image quality after encoding. Specifically, the parameter determination unit 13B determines a reference encoding parameter QP for controlling image quality for each CTU 14 unit within the frame, based on a target encoding amount predetermined by the user and the encoding amount of the bitstream of the image encoded by the encoding unit 11. Hereafter, the reference encoding parameter QP determined by the parameter determination unit 13B will be referred to as "encoding parameter BaseQP".

[0065] The parameter determination unit 13B determines the encoding parameter BaseQP using a known method. The parameter determination unit 13B may determine the encoding parameter BaseQP using a value predetermined by the user, for example. Alternatively, the parameter determination unit 13B may determine the encoding parameter BaseQP by performing code amount control using the R-λ model on the target code amount assigned to the CTU 14, for example, as done in HEVC reference software or VVC reference software.

[0066] The calculation unit 13C calculates the overlap rate between the area of ​​interest 17A determined from the frame by the determination unit 13A and each CTU 14 within the frame.

[0067] Figure 9 shows an example of the positional relationship between Bbox 16, which represents the detection result of a specific object by the detection unit 12, the area of ​​interest 17A determined by the determination unit 13A based on the object detection result, and each CTU 14 within the frame. The numerical values ​​shown within each CTU 14 that overlap with the area of ​​interest 17A in Figure 9 indicate the overlap rate between the area of ​​interest 17A and each CTU 14, calculated by the calculation unit 13C. Note that overlap between CTU 14 and area of ​​interest 17A means that the area of ​​interest 17A overlaps with CTU 14 even slightly.

[0068] Figure 9 shows an example in which the frame, divided into multiple CTUs 14, has overlapping regions of interest 17A based on three Bboxes 16A to 16C, respectively. The region other than the region of interest 17A is the region of non-interest 17B.

[0069] In Figure 9, CTUs 14 that do not overlap with any of the areas of interest 17A are shaded, while CTUs 14 that overlap with any of the areas of interest 17A are not shaded, thereby distinguishing between CTUs 14 that overlap with the areas of interest 17A and CTUs 14 that do not overlap with the areas of interest 17A.

[0070] The calculation unit 13C calculates the overlap rate between the region of interest 17A and each CTU 14 that overlaps with the region of interest 17A. Naturally, the overlap rate for CTU 14 that do not overlap with any of the regions of interest 17A, i.e., the non-region of interest 17B, is "0".

[0071] The modification unit 13D modifies the encoding parameter BaseQP determined by the parameter determination unit 13B according to the degree of overlap between the CTU 14 and the area of ​​interest 17A determined by the determination unit 13A.

[0072] Specifically, the modification unit 13D determines DeltaQP, which is the modification amount of the coding parameter BaseQP, for each CTU 14, and uses the coding parameters BaseQP and DeltaQP to determine the final coding parameter QP to be applied to each CTU 14. As an example, DeltaQP is an integer less than or equal to 0, and the coding parameter QP to be applied to the CTU 14 is calculated by equation (1).

[0073] (Math. 1) QP=BaseQP+DeltaQP...(1)

[0074] The method for correcting the coding parameter QP by the correction unit 13D is not limited to equation (1). For example, if DeltaQP is a weight coefficient represented by a real number between 0 and 1, the coding parameter QP may be calculated by equation (2). In equation (2), the operator "*" means multiplication.

[0075] (Math. 2) QP=BaseQP*DeltaQP...(2)

[0076] The value of DeltaQP is determined according to the degree of overlap between CTU14 and the region of interest 17A, but the method for determining DeltaQP will be explained in detail later.

[0077] For the sake of explanation, this embodiment will describe an example in which the coding parameter QP is calculated using equation (1).

[0078] The image coding device 6, which has these functional units, can be configured using a computer 20. Figure 10 shows an example of the main electrical system configuration of the image coding device 6 configured using a computer 20.

[0079] Computer 20 includes a CPU (Central Processing Unit) 6A, which is an example of a processor that executes the processing of each functional unit in the image encoding device 6 shown in Figure 3. Computer 20 also includes a DRAM 6B, which is used as a temporary workspace for the CPU 6A. Furthermore, computer 20 includes a ROM 6C that stores a startup program for starting the image encoding device 6 when its power is turned on. In addition to the ROM 6C, the image encoding device 6 may also include a storage 8 as shown in Figure 2.

[0080] The CPU 6A causes the computer 20 to function as an image encoding device 6 by executing a program. Specifically, the CPU 6A reads the image encoding program from the storage 8 and executes the image encoding program using the DRAM 6B as a temporary work area. The CPU 6A executes the processing of each functional unit shown in Figure 3 according to the description of the image encoding program.

[0081] The CPU 6A, DRAM 6B, and ROM 6C are connected to each other via bus 6D, and exchange data with each other. Bus 6D is also connected to, for example, an external interface 6E, an input unit 6F, and a memory interface 6G.

[0082] The external I / F 6E is an interface for sending the bitstream generated by the encoding unit 11 and the detection results obtained by the detection unit 12 to the outside.

[0083] The input unit 6F is a unit that receives user instructions and notifies the CPU 6A. Examples of input units 6F include buttons, touch panels, keyboards, and mice.

[0084] Memory I / F 6G is an interface for sending and receiving data to and from storage 8.

[0085] Next, the operation of the image encoding device 6 will be explained. Figure 11 is a flowchart showing an example of the flow of image encoding processing performed by the image encoding device 6 when video is received from the camera 5. The CPU 6A reads the image encoding program stored in the storage 8, loads the read image encoding program into the DRAM 6B, and executes the image encoding processing. The image encoding processing is an example of an image encoding method.

[0086] For the sake of explanation, we will use the image encoding flowchart shown in Figure 11 to describe an example of encoding a single frame that makes up a video. In reality, the image encoding flowchart shown in Figure 11 is repeatedly executed for each frame that makes up the video, thereby generating a bitstream of the encoded video.

[0087] In step S10, the CPU 6A, acting as a preprocessing unit 10, converts the video captured by the camera 5 into a format that can be processed by the encoding unit 11 and the detection unit 12, respectively, and outputs the converted video to the storage 8 via the memory interface 6G.

[0088] In step S20, the CPU 6A, acting as the detection unit 12, detects a specific object from the frames that make up the video whose format has been converted by the processing in step S10. The CPU 6A outputs the detection result to the storage 8 via the memory interface 6G.

[0089] In step S30, the CPU 6A, as a determination unit 13A constituting the control unit 13, uses the detection result obtained in step S20 to determine whether or not the object region detected from the frame is the region of interest 17A.

[0090] In step S40, the CPU 6A, as the encoding unit 11, performs an encoding process that encodes frames in units of CTU 14 according to the encoding parameter QP and outputs the bitstream obtained by the encoding.

[0091] Figure 12 is a flowchart showing an example of the encoding process flow performed by the processing in step S40 of Figure 11. The encoding process shown in Figure 12 is performed once for each CTU 14 that constitutes a frame.

[0092] In step S100, the CPU 6A, as the parameter determination unit 13B of the control unit 13, determines the encoding parameter BaseQP to the encoding parameter QP.

[0093] In step S110, the CPU 6A, as the control unit 13, performs a correction process to modify the encoding parameter QP determined by the processing in step S100.

[0094] Figure 13 is a flowchart showing an example of the flow of the correction process performed by the process in step S110 of Figure 12.

[0095] In step S200 of Figure 13, the CPU 6A, as the calculation unit 13C of the control unit 13, calculates the overlap rate between the region of interest 17A obtained by the processing in step S30 of Figure 11 and the CTU 14.

[0096] In step S210, the CPU 6A, as the modification unit 13D of the control unit 13, determines whether the duplication rate calculated by the processing in step S200 is less than or equal to a predetermined threshold. The threshold for comparison with the duplication rate is, for example, a value set by the user and pre-stored in the ROM 6C or storage 8. The threshold for comparison with the duplication rate can be modified by the user.

[0097] If the overlap rate is below the threshold, the process proceeds to step S220. In step S220, the CPU 6A, as the modification unit 13D of the control unit 13, sets DeltaQP to dQP0 and proceeds to step S240. dQP0 is an integer less than or equal to 0.

[0098] On the other hand, if the overlap rate exceeds the threshold, the process proceeds to step S230. In step S230, the CPU 6A, as the modification unit 13D of the control unit 13, sets DeltaQP to dQP1 and proceeds to step S240. dQP1 is an integer less than or equal to 0, and dQP1 < dQP0.

[0099] In step S240, the CPU 6A, as the modification unit 13D of the control unit 13, modifies the encoding parameter QP according to the above-described equation (1).

[0100] As an example, let's assume the overlap rate threshold is 0.4, dQP0 is 0, and dQP1 is -10. In this case, the coding parameter QP will be "BaseQP" when the overlap rate is 40% or less, and "BaseQP-10" when the overlap rate exceeds 40%. In other words, when the overlap rate exceeds the threshold, a coding parameter QP smaller than the coding parameter QP used when the overlap rate is below the threshold is set.

[0101] In this way, the CPU 6A modifies the encoding parameter BaseQP according to the degree of overlap between the CTU 14 and the region of interest 17A, and generates two encoding parameters QP.

[0102] With the above steps completed, the modification process shown in Figure 13 is finished, and the process proceeds to step S120 shown in Figure 12.

[0103] In step S120 of Figure 12, the CPU 6A, acting as an encoding unit 11, acquires the original image of the CTU 14 to be processed from the storage 8, and also acquires a decoded image obtained by decoding a past original image from the storage 8 as a reference image.

[0104] In step S130, the CPU 6A, acting as the encoding unit 11, determines the division shape and encoding mode of the CTU 14 using the encoding parameter QP that was corrected by the correction process in step S110.

[0105] Figures 14 and 15 show examples of methods for determining the division shape and encoding mode of the CTU 14. Figure 14 shows an example of a method for determining the division shape and encoding mode of the CTU 14 when the overlap rate between the CTU 14 and the region of interest 17A is below a threshold. Figure 15 shows an example of a method for determining the division shape and encoding mode of the CTU 14 when the overlap rate between the CTU 14 and the region of interest 17A exceeds a threshold.

[0106] First, as shown in Figure 14, if the overlap rate between CTU 14 and the region of interest 17A is below a threshold, the CPU 6A performs RD optimization using the coding parameter QP0 to determine the overall division shape and coding mode of CTU 14. The coding parameter QP0 is the coding parameter QP calculated when dQP0 is set as DeltaQP in the correction process shown in Figure 13.

[0107] In this embodiment, RD optimization recursively divides the CTU 14 based on the six division patterns shown in Figure 5, and searches for a combination of division shape and coding mode that minimizes the RD cost calculated when the CU 15 obtained from each division pattern is coded using a modified coding parameter QP in multiple coding modes such as intra-prediction and inter-prediction.

[0108] The RD cost is an example of the value of the cost function related to the disclosed technology, and is calculated as RD cost = D + λ * R. Here, D represents image distortion, R represents the code amount, and λ represents the Lagrange multiplier. Generally, the division shape and coding mode of the CTU14 obtained as a result of RD optimization differ depending on the value of the coding parameter QP.

[0109] Figure 14 shows CTU14-1A after RD optimization of CTU14 using the coding parameter QP0, and after determining the division shape and coding mode of CTU14. In Figure 14, CU15 to which the coding parameter QP0 is applied is not shaded.

[0110] CPU 6A applies the encoding parameter QP1 to the CU 15 that overlaps with the region of interest 17A of CTU 14-1A (hereinafter referred to as the "first overlapping CU 15"). The encoding parameter QP1 is the encoding parameter QP calculated when dQP1 is set as DeltaQP in the correction process shown in Figure 13. Furthermore, overlap between CU 15 and the region of interest 17A means that the region of interest 17A overlaps with CU 15 even slightly.

[0111] Furthermore, in relation to the first overlapping CU15, a CU15 that does not overlap with the area of ​​interest 17A is called the "first non-overlapping CU15".

[0112] CTU14-1B in Figure 14 represents the state in which the encoding parameter QP1 is applied to the first duplicate CU15. In Figure 14, the CU15 to which the encoding parameter QP1 is applied is shaded.

[0113] The CPU 6A performs RD optimization of the first duplicate CU 15 using the encoding parameter QP1, and re-determines the division shape and encoding mode within the first duplicate CU 15.

[0114] Figure 14 shows the state of CTU14-1C after the division shape and encoding mode of the first duplicate CU15 have been re-determined.

[0115] As a result of re-determining the division shape and encoding mode for the first duplicate CU 15, a first non-duplicate CU 15 may occur within the range of the first duplicate CU 15 before re-determination. The CPU 6A applies the encoding parameter QP0 to the first non-duplicate CU 15 that occurred as a result of re-determining the division shape and encoding mode.

[0116] Figure 14 shows CTU14-1D, which represents the state after applying the encoding parameter QP0 to the newly generated first non-duplicate CU15.

[0117] The CPU 6A performs RD optimization on the first non-duplicate CU 15 generated as a result of RD optimization of the first duplicate CU 15, using the encoding parameter QP0, and determines the division shape and encoding mode within the range of the first non-duplicate CU 15.

[0118] Figure 14 shows that CTU14-1E represents the final segmentation shape and encoding mode of CTU14.

[0119] In other words, if the overlap rate between the CTU 14 and the region of interest 17A is below a threshold, the CPU 6A divides the CTU 14 into CU 15 according to the division shape and encoding mode of the CTU 14 based on the encoding parameter QP0, and applies the encoding parameter QP1 to the first overlapping CU 15 to redetermine the division shape and encoding mode of the first overlapping CU 15 of the CTU 14.

[0120] On the other hand, as shown in Figure 15, if the overlap rate between CTU 14 and the region of interest 17A exceeds a threshold, the CPU 6A performs RD optimization using the coding parameter QP1 to determine the overall division shape and coding mode of CTU 14.

[0121] Figure 15 shows CTU14-2A, which represents the state after RD optimization of CTU14 using the coding parameter QP1 to determine the division shape and coding mode of CTU14. In Figure 15, the CU15 to which the coding parameter QP1 has been applied is shaded.

[0122] The CPU 6A performs RD optimization on the CTU 14 using the encoding parameter QP1, and applies the encoding parameter QP0 to the first non-overlapping CU 15 that is generated as a result of determining the division shape and encoding mode.

[0123] Figure 15 shows CTU14-2B, which represents the state where the encoding parameter QP0 is applied to the first non-duplicate CU15. In Figure 15, the CU15 to which the encoding parameter QP0 is applied is not shaded.

[0124] The CPU 6A performs RD optimization of the first non-overlapping CU 15 using the encoding parameter QP0, and re-determines the partition shape and encoding mode within the first non-overlapping CU 15.

[0125] Figure 15 shows the state of CTU14-2C after the division shape and encoding mode of the first non-overlapping CU15 have been re-determined.

[0126] In other words, if the overlap rate between the CTU 14 and the region of interest 17A exceeds a threshold, the CPU 6A divides the CTU 14 into CU 15 according to the division shape and encoding mode of the CTU 14 based on the encoding parameter QP1, and applies the encoding parameter QP0 to the first non-overlapping CU 15 to re-determine the division shape and encoding mode of the first non-overlapping CU 15 of the CTU 14.

[0127] In this way, the image encoding device 6 sets encoding parameter QP0 and encoding parameter QP1 which is smaller than encoding parameter QP0, and for each CTU 14, it switches the encoding parameter QP used for RD optimization so that for the non-focus region 17B, an RD-optimized division shape and encoding mode are obtained using encoding parameter QP0, and for the focus region 17A, an RD-optimized division shape and encoding mode are obtained using encoding parameter QP1.

[0128] Setting the encoding parameter QP lower increases the amount of code but improves image quality. Conversely, setting the encoding parameter QP higher decreases the amount of code but worsens image quality. Therefore, the CPU 6A can reduce the overall code amount of the image while relatively improving the image quality of the area of ​​interest 17A by lowering the image quality of the image area corresponding to the non-area of ​​interest 17B compared to the image area corresponding to the area of ​​interest 17A.

[0129] After determining the division shape and encoding mode of the CTU 14 in this manner, in step S140 of Figure 12, the CPU 6A, acting as the encoding unit 11, encodes the CTU 14 and outputs a bitstream. With this, the encoding process shown in Figure 12 is completed, as is the image encoding process shown in Figure 11.

[0130] As a result, the image encoding device 6 can reduce the network's video transmission bandwidth while maintaining the inference accuracy in the cloud 3 using the encoded video.

[0131] <Modification of Encoding Process> In the embodiment described above, the image quality of the image region corresponding to the non-focus region 17B was reduced compared to the image quality of the image region corresponding to the focus region 17A. However, in some cases, it may be desirable to deliberately reduce the image quality of the image region corresponding to the focus region 17A. For example, in applications where it is desirable to blur the faces of people detected from the video for privacy protection, it may be desirable to deliberately reduce the image quality of the image region corresponding to the focus region 17A, which includes the faces of people, compared to the image quality of the image region corresponding to the non-focus region 17B.

[0132] Therefore, in this modification, an encoding process is described in which the image quality of the image region corresponding to the area of ​​interest 17A is reduced to that of the image region corresponding to the area not of interest 17B.

[0133] In this modified example, the encoding process differs only in step S130 of the encoding process shown in Figure 12; all other processes are the same as those shown in Figure 12. Therefore, the process performed in step S130 of Figure 12 will be explained.

[0134] Figures 16 and 17 show modified examples of the determination method for determining the division shape and encoding mode of the CTU 14. Figure 16 shows, for example, an example of the method for determining the division shape and encoding mode of the CTU 14 when the overlap rate between the CTU 14 and the region of interest 17A is below a threshold. Figure 17 shows, for example, an example of the method for determining the division shape and encoding mode of the CTU 14 when the overlap rate between the CTU 14 and the region of interest 17A exceeds a threshold.

[0135] As shown in Figure 16, if the overlap rate between CTU 14 and the region of interest 17A is below a threshold, the CPU 6A performs RD optimization using the coding parameter QP1 to determine the overall division shape and coding mode of CTU 14.

[0136] Figure 16 shows CTU14-3A, which represents the state after RD optimization of CTU14 using the coding parameter QP1 to determine the division shape and coding mode of CTU14. In Figure 16, CU15 to which the coding parameter QP1 is applied is shaded.

[0137] The CPU 6A performs RD optimization on the CTU 14 using the encoding parameter QP1 to determine the division shape and encoding mode, and applies the encoding parameter QP0 to the resulting CU 15 (hereinafter referred to as "second non-overlapping CU 15") that does not overlap with the non-focus region 17B.

[0138] Figure 16 shows CTU14-3B, which represents the state where the encoding parameter QP0 is applied to the second non-overlapping CU15. In Figure 16, the CU15 to which the encoding parameter QP0 is applied is not shaded.

[0139] The CPU 6A performs RD optimization of the second non-overlapping CU 15 using the encoding parameter QP0, and re-determines the partition shape and encoding mode within the second non-overlapping CU 15.

[0140] Figure 16 shows the state of CTU14-3C after the division shape and encoding mode of the second non-overlapping CU15 have been re-determined.

[0141] In other words, if the overlap rate between the CTU 14 and the region of interest 17A is below a threshold, the CPU 6A divides the CTU 14 into CU 15 according to the division shape and encoding mode of the CTU 14 based on the encoding parameter QP1, and applies the encoding parameter QP0 to the second non-overlapping CU 15 to re-determine the division shape and encoding mode of the second non-overlapping CU 15 of the CTU 14.

[0142] On the other hand, as shown in Figure 17, if the overlap rate between CTU 14 and the region of interest 17A exceeds a threshold, CPU 6A performs RD optimization using the coding parameter QP0 to determine the overall division shape and coding mode of CTU 14.

[0143] Figure 17 shows CTU14-4A after RD optimization of CTU14 using the coding parameter QP0, determining the division shape and coding mode of CTU14. In Figure 17, CU15 to which the coding parameter QP0 is applied is not shaded.

[0144] CPU 6A applies the encoding parameter QP1 to the CU 15 that overlaps with the non-focus region 17B of CTU 14-4A (hereinafter referred to as "second overlapping CU 15").

[0145] Figure 17 shows CTU14-4B, which represents the state where the encoding parameter QP1 is applied to the second duplicate CU15. In Figure 17, the CU15 to which the encoding parameter QP1 is applied is shaded.

[0146] The CPU 6A performs RD optimization of the second duplicate CU 15 using the encoding parameter QP1, and re-determines the division shape and encoding mode within the second duplicate CU 15.

[0147] Figure 17 shows the state of CTU14-4C after the division shape and encoding mode of the second duplicate CU15 have been re-determined.

[0148] As a result of re-determining the division shape and encoding mode for the second duplicate CU15, a second non-duplicate CU15 may occur within the range of the second duplicate CU15 before re-determination. The CPU 6A applies the encoding parameter QP0 to the second non-duplicate CU15 that occurred as a result of re-determining the division shape and encoding mode.

[0149] Figure 17 shows CTU14-4D, which represents the state after applying the encoding parameter QP0 to the newly generated second non-duplicate CU15.

[0150] The CPU 6A performs RD optimization on the second non-duplicate CU 15 that was generated as a result of RD optimization of the second duplicate CU 15, using the encoding parameter QP0, and determines the division shape and encoding mode within the range of the second non-duplicate CU 15.

[0151] Figure 17 shows that CTU14-4E represents the final division shape and encoding mode of CTU14.

[0152] In other words, if the overlap rate between the CTU 14 and the region of interest 17A exceeds a threshold, the CPU 6A divides the CTU 14 into CU 15 according to the division shape and encoding mode of the CTU 14 based on the encoding parameter QP0, and applies the encoding parameter QP1 to the second overlapping CU 15 to re-determine the division shape and encoding mode of the second overlapping CU 15 of the CTU 14.

[0153] Thus, the image encoding device 6 may set an encoding parameter QP0 and an encoding parameter QP1 that is smaller than the encoding parameter QP0, and for each CTU 14, it may switch the encoding parameter QP used for RD optimization so that for the region of interest 17A, an RD-optimized division shape and encoding mode are obtained using the encoding parameter QP0, and for the region of non-interest 17B, an RD-optimized division shape and encoding mode are obtained using the encoding parameter QP1.

[0154] The CPU 6A can reduce the overall code size of the image while relatively improving the image quality of the non-focus area 17B by lowering the image quality of the image area corresponding to the focus area 17A compared to the image area corresponding to the non-focus area 17B.

[0155] As a result, the image encoding device 6 can reduce the network's video transmission bandwidth while maintaining the inference accuracy in the cloud 3 using the encoded video.

[0156] Although one embodiment of the image encoding device 6 has been described above using the embodiments, the disclosed embodiment of the image encoding device 6 is merely an example, and the embodiment of the image encoding device 6 is not limited to the scope described in the embodiments. Various modifications or improvements can be made to the embodiments without departing from the gist of this disclosure, and such modified or improved embodiments are also included in the technical scope of the disclosure.

[0157] In the above embodiment, as an example, a configuration in which each process shown in Figures 11 to 13 is implemented in software was described. However, it is also possible to have the processes equivalent to the flowcharts for each process be processed in hardware. In this case, the processing speed can be increased compared to when each process is implemented in software.

[0158] In the above embodiment, the processes that the CPU 6A reads and executes software, i.e., programs, may be executed by various processors other than the CPU 6A. Examples of such processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to execute particular processes, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit configurations can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits). Furthermore, each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0159] Furthermore, although the above embodiment describes an embodiment in which the image encoding program is pre-installed on the storage 8, the invention is not limited to this. The image encoding program may be provided in a form stored on a non-transitor storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), Blu-ray, and USB (Universal Serial Bus) memory. In addition, the observation data prediction program may be provided in a form downloaded from an external device via a communication line. This disclosure is also applicable to programs and program products.

[0160] The following additional information is disclosed regarding the embodiments described above.

[0161] (Note 1) An image encoding device comprising: a determination unit that determines whether an object region detected from an image is a region of interest that includes a target for processing; an encoding parameter determination unit that determines encoding parameters for controlling the image quality after encoding; a modification unit that modifies the encoding parameters determined by the encoding parameter determination unit according to the degree of overlap between the CTU, which is a divided image of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the region of interest determined by the determination unit; and an encoding unit that determines an encoding mode, which is a group of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTU, so as to minimize the value of a predetermined cost function calculated in conjunction with encoding according to the encoding parameters modified by the modification unit, and encodes the image for each CTU using the determined division shape of the CTU and the encoding mode.

[0162] (Note 2) The image encoding apparatus according to Note 1, wherein the modification unit modifies the encoding parameters such that the value of the first encoding parameter, which is an encoding parameter used for encoding the CTU whose degree of overlap is less than a predetermined threshold, becomes smaller than the value of the second encoding parameter, which is an encoding parameter used for encoding the CTU whose degree of overlap is greater than or equal to the threshold; the encoding unit divides the CTU into sub-regions according to the division shape of the CTU and the encoding mode based on the first encoding parameter; applies the second encoding parameter to the overlapping sub-region of the CTU, which is a sub-region of the CTU that overlaps with the region of interest, to re-determine the division shape and the encoding mode of the overlapping sub-region of the CTU; and if, as a result of dividing the overlapping sub-region of the CTU based on the re-determined division shape and the encoding mode of the overlapping sub-region of the CTU, a sub-region that no longer overlaps with the region of interest is generated in the overlapping sub-region of the CTU, the first encoding parameter is applied to the sub-region that no longer overlaps with the region of interest to re-determine the division shape and the encoding mode of the sub-region that no longer overlaps with the region of interest.

[0163] (Note 3) The image encoding apparatus according to Note 1, wherein the modification unit modifies the encoding parameters such that the value of the first encoding parameter, which is an encoding parameter used for encoding the CTU whose degree of overlap is less than a predetermined threshold, is smaller than the value of the second encoding parameter, which is an encoding parameter used for encoding the CTU whose degree of overlap is greater than or equal to the threshold; the encoding unit divides the CTU into partial regions according to the division shape of the CTU and the encoding mode based on the second encoding parameter, and applies the first encoding parameter to the non-overlapping partial regions of the CTU, which are partial regions of the CTU that do not overlap with the region of interest, to redetermine the division shape of the non-overlapping partial regions of the CTU.

[0164] (Note 4) The image encoding apparatus according to Note 1, wherein the modification unit modifies the encoding parameters such that the value of the first encoding parameter, which is an encoding parameter used for encoding the CTU whose degree of overlap is less than a predetermined threshold, is smaller than the value of the second encoding parameter, which is an encoding parameter used for encoding the CTU whose degree of overlap is greater than or equal to the threshold; the encoding unit divides the CTU into sub-regions according to the division shape of the CTU and the encoding mode based on the second encoding parameter; and as a result of the division, the first encoding parameter is applied to the non-overlapping sub-regions of the CTU, which are sub-regions of the CTU that no longer overlap with the non-attention region, which is a region other than the area of ​​interest, to redetermine the division shape of the non-overlapping sub-regions of the CTU.

[0165] (Note 5) The modification unit modifies the coding parameters such that the value of the first coding parameter, which is the coding parameter used for coding the CTU whose degree of overlap is less than a predetermined threshold, becomes smaller than the value of the second coding parameter, which is the coding parameter used for coding the CTU whose degree of overlap is equal to or greater than the threshold. The image encoding apparatus according to Appendix 1, wherein the encoding unit divides the CTU into sub-regions according to the division shape of the CTU and the encoding mode based on the first encoding parameter, applies the second encoding parameter to the overlapping sub-region of the CTU which is a sub-region of the CTU that overlaps with a non-focus region which is a region other than the focus region, to redetermine the division shape and the encoding mode of the overlapping sub-region of the CTU, and as a result of dividing the overlapping sub-region of the CTU based on the redetermined division shape and the encoding mode of the overlapping sub-region of the CTU, if a sub-region that no longer overlaps with the non-focus region is generated in the overlapping sub-region of the CTU, the first encoding parameter is applied to the sub-region that no longer overlaps with the non-focus region to redetermine the division shape and the encoding mode of the sub-region that no longer overlaps with the non-focus region.

[0166] (Note 6) An image encoding method in which a computer performs the following steps: determine whether an object region detected from an image is a region of interest that includes the object to be processed; determine encoding parameters to control the image quality after encoding; modify the determined encoding parameters according to the degree of overlap between the CTUs, which are divided images of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the determined region of interest; determine an encoding mode, which is a set of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTUs, so as to minimize the value of a predetermined cost function calculated in conjunction with encoding according to the modified encoding parameters; and encode the image for each CTU using the determined division shape of the CTUs and the encoding mode.

[0167] (Note 7) An image encoding program that causes a computer to perform the following process: determine whether an object region detected from an image is a region of interest that includes the object to be processed; determine encoding parameters to control the image quality after encoding; modify the determined encoding parameters according to the degree of overlap between the CTUs, which are the divided images of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the determined region of interest; determine an encoding mode, which is a set of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTUs, so as to minimize the value of a predetermined cost function calculated in conjunction with encoding according to the modified encoding parameters; and encode the image for each CTU using the determined division shape of the CTUs and the encoding mode.

[0168] (Note 8) A computer program product that causes a computer to perform the following process: determine whether an object region detected from an image is a region of interest that includes the object to be processed; determine encoding parameters to control the image quality after encoding; modify the determined encoding parameters according to the degree of overlap between the CTUs, which are the divided images of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the determined region of interest; determine an encoding mode, which is a set of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTUs, so as to minimize the value of a predetermined cost function calculated in conjunction with encoding according to the modified encoding parameters; and encode the image for each CTU using the determined division shape of the CTUs and the encoding mode.

[0169] (Note 9) A non-temporary storage medium storing a program executable by a computer to perform image coding processing, the image coding processing comprising: determining whether an object region detected from an image is a region of interest that includes a target for processing; determining coding parameters to control the image quality after coding; modifying the determined coding parameters according to the degree of overlap between the CTU, which is a divided image of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the determined region of interest; determining a coding mode, which is a group of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTU, such that the value of a predetermined cost function calculated in conjunction with coding according to the modified coding parameters is minimized, and coding the image for each CTU using the determined division shape of the CTU and the coding mode.

[0170] (Note 10) An image encoding device comprising a processor, the processor determines whether an object region detected from an image is a region of interest that includes a target for processing, determines encoding parameters to control the image quality after encoding, modifies the determined encoding parameters according to the degree of overlap between the determined region of interest and the CTU, which is a divided image of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the determined region of interest, determines an encoding mode, which is a group of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTU, so as to minimize the value of a predetermined cost function calculated in conjunction with encoding according to the modified encoding parameters, and encodes the image for each CTU using the determined division shape of the CTU and the encoding mode.

Claims

A determination unit that determines whether or not the object region detected from the image is a region of interest that includes the object to be processed, A parameter determination unit that determines encoding parameters to control the image quality of the encoded image, A modification unit modifies the encoding parameters determined by the parameter determination unit according to the degree of overlap between the CTU, which is a divided image of the image obtained by dividing the image into a plurality of blocks according to a predetermined division rule, and the region of interest determined by the determination unit. An encoding unit that determines an encoding mode, which is a set of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTU, so as to minimize the value of a predetermined cost function calculated by encoding according to the encoding parameters corrected by the correction unit, and encodes the image for each CTU using the determined division shape of the CTU and the encoding mode, An image encoding device equipped with an image encoding device. 　 The modification unit modifies the encoding parameters determined by the parameter determination unit according to the degree of overlap between the CTU and the region of interest, thereby generating two encoding parameters. The encoding unit divides the CTU into sub-regions according to the division shape of the CTU and the encoding mode based on one encoding parameter, and applies the other encoding parameter to the sub-region of the CTU that overlaps with the region of interest, or to the sub-region of the CTU that does not overlap with the non-region of interest which is a region other than the region of interest, thereby redetermining the division shape and the encoding mode of the sub-region of the CTU. The image encoding apparatus according to claim 1. 　 The modification unit modifies the encoding parameters determined by the parameter determination unit according to the degree of overlap between the CTU and the region of interest, thereby generating two encoding parameters. The encoding unit divides the CTU into sub-regions according to the division shape of the CTU and the encoding mode based on one encoding parameter, and applies the other encoding parameter to the sub-regions of the CTU that do not overlap with the region of interest, or to the sub-regions of the CTU that overlap with the non-region of interest which is a region other than the region of interest, thereby redetermining the division shape and the encoding mode of the sub-regions of the CTU. The image encoding apparatus according to claim 1. 　 Determine whether the object region detected from the image is the region of interest that includes the object to be processed. Determine the encoding parameters that control the image quality after encoding. The encoding parameters determined are modified according to the degree of overlap between the CTU, which is a divided image of the image obtained by dividing the image into multiple blocks according to a predetermined division rule, and the determined region of interest. The computer determines an encoding mode, which is a set of parameters used to generate a predicted image of the image from the quantized image, and the division shape of the CTU, so as to minimize the value of a predetermined cost function calculated when encoding according to the modified encoding parameters. The computer then performs a process of encoding the image for each CTU using the determined division shape of the CTU and the encoding mode. Image encoding method.

Images