Image encoding device and image encoding method

Abstract

This image encoding device comprises: an object tracking unit that tracks an object in each frame constituting a video and outputs tracking information including the degree of success or failure in tracking the object from the past to the present; a decision unit that decides whether each region in the frame is a region of interest, on the basis of the tracking information; a determination unit that determines an encoding parameter for controlling image quality for each region in the frame, on the basis of a result of the decision; a correction unit that corrects the encoding parameter on the basis of the tracking information; and an encoding unit that performs encoding by using the encoding parameter for each region in the frame.

Description

Image encoding device and image encoding method

[0001] The technology disclosed herein relates to an image coding device and an image coding method.

[0002] The need for AI (Artificial Intelligence)-based video analysis is increasing, and it is expected to have applications in various fields such as autonomous driving, surveillance, and monitoring of vehicles. In recent years, AI processing systems that collaborate between edge terminals and the cloud have attracted attention, such as transmitting video from edge terminals to the cloud and performing video monitoring and advanced video analysis on the cloud. In such AI processing systems, the cost of video transmission when transmitting video from edge terminals to the cloud remains a challenge.

[0003] As a method that enables high-quality video transmission while reducing video transmission costs, ROI coding technology is known, which applies different video encoding processing to the region of interest (ROI) and other areas (non-interest regions) within each frame of a video. In ROI coding technology, low-compression, high-quality video encoding is performed for the region of interest, while high-compression, low-quality video encoding is performed for the non-interest regions. This aims to achieve both a reduction in the video transmission bitrate and preservation of the image quality of the region of interest.

[0004] For example, Non-Patent Document 1 describes using ROI coding technology to encode important regions for AI processing performed after video encoding with low compression and high image quality. Specifically, important regions are detected using object detectors such as YOLO (You Only Look Once) to obtain an object detection frame (hereinafter referred to as Bbox (BoundingBox)). Then, using coding blocks called CTU (Coding Tree Unit) as units, the regions of interest are identified depending on whether or not they overlap with the Bbox. Subsequently, the quantization parameters (QP: Quantization Parameter) of the identified regions of interest are uniformly reduced compared to the non-regions of interest. This achieves both a reduction in the bitrate of video transmission and suppression of the degradation of the inference accuracy of AI processing on the decoded video on the cloud.

[0005] Thus, in ROI coding technology, the region of interest is identified based on Bboxes detected by a high-precision AI-based object detector. Here, Bboxes with a confidence score above a predetermined threshold are used to identify the region of interest. The confidence score is output from the object detector along with the Bbox, and its value is, for example, a real number between 0.0 and 1.0. As the threshold approaches 0.0, it becomes easier to detect objects, but false detections increase. As the threshold approaches 1.0, false detections can be reduced, but it becomes more difficult to detect objects. Therefore, the threshold is generally determined in advance by the user, taking into account the trade-off between the ease of object detection and the likelihood of false detections.

[0006] Kristian Fischer, etc. “Saliency-Driven Versatile Video Coding for Neural Object Detection,” 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021

[0007] Because the confidence score of Bbox can change from frame to frame, it can be difficult to continuously and stably detect the same object, especially in Bboxes where the confidence score is near the threshold. In other words, the identification results of the region of interest and the region of non-interest may fluctuate from frame to frame, and consequently, the QP may also fluctuate significantly from frame to frame. This fluctuation in QP may impair the subjective and objective image quality of the region of interest. Furthermore, this problem can occur not only in AI-based object detectors, but also in classical ROI coding techniques that use thresholding on feature quantities such as color, brightness, and complexity of images.

[0008] The disclosed technology was developed in view of the above points, and aims to provide an image encoding device and method that can suppress changes in image quality from frame to frame.

[0009] A first aspect of the present disclosure is an image encoding device comprising: an object tracking unit that tracks an object in each frame constituting an image and outputs tracking information including the degree to which the tracking of the object has been successful or unsuccessful from the past to the present; a determination unit that determines whether each region in the frame is a region of interest based on the tracking information; a determination unit that determines encoding parameters for controlling image quality for each region in the frame based on the result of the determination; a modification unit that modifies the encoding parameters based on the tracking information; and an encoding unit that encodes each region in the frame using the encoding parameters.

[0010] A second aspect of the present disclosure is an image encoding method, wherein a computer performs the following processes: tracking an object in each frame constituting an image; outputting tracking information including the degree to which the tracking of the object has been successful or unsuccessful from the past to the present; determining whether each region within the frame is a region of interest based on the tracking information; determining encoding parameters for controlling image quality for each region within the frame based on the result of the determination; modifying the encoding parameters based on the tracking information; and encoding each region within the frame using the encoding parameters.

[0011] According to the disclosed technology, frame-to-frame changes in image quality can be suppressed.

[0012] This figure shows an example of the configuration of an event-driven inference system. This block shows an example of the hardware configuration of an image encoding device. This block shows an example of the functional configuration of an image encoding device. This block shows an example of the detailed functional configuration of the tracking unit. This figure shows an example of the positional relationship between Bbox, the region of interest, and the CTU. This figure shows an example of a function that derives the change in the amount of modification of the encoding parameters. This figure shows an example of a function that derives the change in the amount of modification of the encoding parameters. This figure shows an example of the configuration of a table that holds the amount of modification of the encoding parameters. This figure shows an example of the change in the amount of modification of the encoding parameters for each frame. This flowchart shows an example of the flow of the image encoding process. This flowchart shows an example of the flow of the tracking process. This figure shows an example of the configuration of a table that holds the amount of modification of the encoding parameters.

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0014] [First Embodiment] Figure 1 shows an example of the configuration of an event-driven inference system 1. The event-driven inference system 1 comprises a cloud 3 and at least one edge terminal 4. In the event-driven inference system 1, when the edge terminal 4 detects an event, it transmits video to the cloud 3, and the cloud 3 performs various inference processing on the received video. In this embodiment, "video" refers to a moving image formed by sequentially connecting multiple images (still images) in chronological order. In addition, individual still images that make up the video (moving image) may be simply called "images" or "frames".

[0015] Cloud 3 is a form of cloud computing, built in a data center or similar location. Cloud 3 receives video from edge terminals 4 and performs predetermined inference processing on the received video. The inference results obtained from Cloud 3's inference processing are used by the collaborative server 2, which runs other services. Therefore, Cloud 3 transmits the inference results to the collaborative server 2 via the network. Furthermore, Cloud 3 can control the operation of edge terminals 4 by sending instructions to edge terminals 4 according to the obtained inference results.

[0016] The edge terminal 4 is connected to the cloud 3 wirelessly via a network. The edge terminal 4 includes a video capture device. The edge terminal 4 also detects events based on the video captured by the capture device and transmits the video to the cloud 3 at the time the event is detected.

[0017] The edge terminal 4 can be any type of terminal suitable for the content of the events to be monitored and the content to be inferred in the cloud 3, and there are no restrictions on the type of edge terminal 4. For example, edge terminals 4 include surveillance cameras, drones, automobiles, and mobile terminals. In this embodiment, when describing surveillance cameras, drones, automobiles, and mobile terminals without distinction, the general term "edge terminal 4" is used. Furthermore, there are no restrictions on the number of edge terminals 4 included in the event-driven inference system 1.

[0018] In artificial intelligence (AI) processing systems such as event-driven inference system 1, it is desirable to enable high-quality video transmission while reducing video transmission costs when transmitting video from edge terminal 4 to cloud 3. As a method to achieve this, ROI coding technology is known, which processes video encoding differently for the region of interest (ROI) and other areas (non-interest regions) within each frame of video. ROI coding technology performs low-compression, high-quality video encoding for the region of interest and high-compression, low-quality video encoding for the non-interest regions. This aims to achieve both a reduction in the video transmission bitrate and preservation of image quality in the region of interest.

[0019] The edge terminal 4 is equipped with an image encoding device 10 for suppressing frame-to-frame image quality changes in relation to such ROI encoding technology. The image encoding device 10 is a device for performing predetermined processing, including image encoding, on video. The image encoding device 10 according to this embodiment will be described below.

[0020] Figure 2 is a block diagram showing an example of the hardware configuration of the image encoding device 10. The image encoding device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, storage 14, an input unit 15, a display unit 16, a communication interface (Interface) 17, and a camera 18. Each component is connected to the others via a bus 19 so as to be able to communicate with each other.

[0021] The CPU 11 is a central processing unit that executes various programs and controls various parts. Specifically, the CPU 11 reads a program from the ROM 12 or storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculations according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores an image encoding program for executing the image encoding process described later. The image encoding program may be a single program or a group of programs composed of multiple programs or modules.

[0022] ROM 12 stores various programs and data. RAM 13 temporarily stores programs or data as a working area. Storage 14 consists of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive) and stores various programs and data, including the operating system.

[0023] The input unit 15 includes a pointing device such as a mouse, a keyboard, and buttons, and is used for various types of input. The display unit 16 is, for example, a liquid crystal display and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0024] Communication I / F17 is an interface for communicating with other devices, including Cloud 3. For this communication, wired communication standards such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface), or wireless communication standards such as 4G, 5G, or Wi-Fi (registered trademark) may be used.

[0025] Camera 18 is an example of a shooting device for capturing images. Camera 18 has an image sensor and an imaging optical system including a lens that forms an image of a subject on the imaging surface of the image sensor. The image sensor is, for example, an image sensor capable of capturing color images, such as a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0026] Next, the functional configuration of the image encoding device 10 will be described. Figure 3 is a block diagram showing an example of the functional configuration of the image encoding device 10. The image encoding device 10 comprises a preprocessing unit 20, an encoding unit 22, an object tracking unit 30, and a control unit 40. Each functional configuration is realized by the CPU 11 reading an image encoding program stored in the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Note that the preprocessing unit 20, the encoding unit 22, and the object tracking unit 30 may each be realized by a dedicated accelerator.

[0027] The preprocessing unit 20 converts the video input from the camera 18 into a format that the encoding unit 22 can process, and stores the converted video in the storage unit 14. As an example, the preprocessing unit 20 converts the video to the YUV420 format.

[0028] Furthermore, the preprocessing unit 10 converts the video input from the camera 18 into a format that the object tracking unit 30 can process, and stores the converted video in the storage unit 14. As an example, the preprocessing unit 20 converts the video to RGB format.

[0029] The encoding unit 22 encodes each region, which is an encoding block within a frame, using encoding parameters for controlling the image quality after encoding, which are determined and modified by the control unit 40, described later. For example, a CTU (Coding Tree Unit) and a CU (Coding Unit) can be applied as the region unit, which is an encoding block.

[0030] Specifically, the encoding unit 22 compresses and encodes the video using a lossy encoding method. The encoding unit 22 is a so-called encoder. The encoding unit 22 reads the video stored in the storage 14 by the preprocessing unit 20, performs compression encoding in accordance with the video encoding standard, and stores the bitstream obtained by the compression encoding in the storage 14. As video encoding standards, for example, H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding) can be applied as appropriate.

[0031] For the sake of explanation, the image before encoding by the encoding unit 22 is sometimes referred to as the "original image." In other words, the image that makes up the video whose format has been converted by the preprocessing unit 20 is an example of an original image.

[0032] The object tracking unit 30 outputs tracking information for each object by tracking objects within each frame that makes up the video. The tracking information includes geometric information such as the object's position, size, velocity, and variations thereof. The tracking information also includes the degree to which object tracking has been successful or unsuccessful from the past to the present. Hereafter, this parameter indicating the degree will be called the "tracking degree parameter". The tracking information may also include an ID assigned to uniquely identify the object.

[0033] For example, the tracking information may include information indicating the duration for which object detection is ongoing (i.e., the duration for which object tracking has been successful) as a tracking degree parameter. In this embodiment, this duration is represented by the number of frames, and this number of frames is called "hit_streak". For example, "hit_streak" is a counter that is incremented by +1 when object tracking is successful and cleared to "0" when tracking fails. Note that if tracking information is generated based on the detection results of an object newly detected from the current frame, "hit_streak" will be "0".

[0034] For example, tracking information may include a tracking degree parameter (flag) indicating whether or not the tracking information was predicted from past tracking information. In this embodiment, this flag indicating whether or not a prediction was made is called "predict_flag". If "predict_flag" is "True", it indicates that the tracking information was predicted. That is, it means that tracking of the object failed in the current frame, and therefore the current tracking information was predicted based on past tracking information. On the other hand, if "predict_flag" is "False", it indicates that the tracking information was not predicted. This means that tracking of the object was successful, and the tracking information was updated based on the detection result from the current frame. Note that if tracking information is generated based on the detection result of an object newly detected from the current frame, "predict_flag" will also be "False".

[0035] For example, the tracking information may also include information indicating the period since the tracking information was last updated (i.e., the period during which object tracking failed and only prediction continued) as a tracking degree parameter. In this embodiment, this period is represented by the number of frames, and this number of frames is called "time_since_update". For example, "time_since_update" is a counter that is cleared to "0" when object tracking is successful and the tracking information is updated, and incremented by +1 when object tracking fails and the tracking information is not updated (i.e., only prediction was performed). Note that if tracking information is generated based on the detection result of an object newly detected from the current frame, "time_since_update" will be "0".

[0036] As methods for generating tracking information, known object tracking methods such as SORT (Simple Online Realtime Tracking) and FairMOT (on the Fairness of detection and reidentification in Multiple Object Tracking) can be used. Recent object tracking methods include an approach called Tracking-by-Detection, which divides and executes two tasks: "the process of detecting an object" and "the process of tracking the detected object," and an approach called End-to-End, which executes the above two tasks simultaneously.

[0037] As an example, the object tracking unit 30 according to this embodiment generates tracking information by dividing it into two tasks, "processing for detecting objects" and "processing for tracking detected objects," similar to SORT. Specifically, the object tracking unit 30 includes a detection unit 32 that performs "processing for detecting objects" and a tracking unit 34 that performs "processing for tracking detected objects."

[0038] The detection unit 32 detects specific objects within each frame that makes up the video. Specifically, the detection unit 32 performs object detection processing to detect predetermined objects (e.g., people and cars) from the original image and outputs the detection result. The detection result is, for example, an object detection frame (hereinafter referred to as "Bbox" (BoundingBox)) that indicates the position and size of a specific object in the original image. The detection unit 32 is, for example, an AI inference core capable of executing an object detection model that has been pre-trained by a neural network.

[0039] Furthermore, Bbox may be accompanied by a confidence score, which is information indicating the confidence level of the object detection result. The confidence score is output from the object detector and attached to Bbox, and takes a value corresponding to the confidence level of the detection result, which is, for example, a real number between 0.0 and 1.0.

[0040] The tracking unit 34 tracks each object between frames based on the detection result (Bbox) of each object and outputs tracking information for each object. In other words, the tracking unit 34 outputs tracking information for each object for each frame. Details of the tracking unit 34 will be described later.

[0041] The control unit 40 determines the area of ​​interest within each frame of the video based on tracking information and determines encoding parameters for each area so that the image quality differs between the area of ​​interest and the area not of interest. The control unit 40 also modifies the encoding parameters based on tracking information so that the encoding parameters do not fluctuate significantly from frame to frame. Details of the control unit 40 will be described later.

[0042] Furthermore, the control unit 40 reads the bitstream stored in the storage 14 when it detects an event and transmits the bitstream to the cloud 3 via the communication interface 17. The control unit 40 may also transmit the detection result (Bbox, etc.) from the detection unit 32 corresponding to the bitstream to the cloud 3 along with the bitstream.

[0043] [Details of the Tracking Unit 34] The detailed functional configuration of the tracking unit 34 will be described with reference to Figure 4. The tracking unit 34 comprises a prediction unit 35, an allocation unit 36, and an update unit 37. As an example, the tracking unit 34 models and tracks the movement of each object using a discrete-time linear Kalman filter (hereinafter referred to as "Kalman filter"), similar to the method used in SORT.

[0044] The Kalman filter algorithm is generally divided into a prediction phase and an update phase. In the prediction phase, the state vector at the current time is predicted based on the previously estimated state vector. In the update phase, the predicted state vector is updated from the obtained observed values. The prediction unit 35 performs processing equivalent to the prediction phase of the Kalman filter, and the update unit 37 performs processing equivalent to the update phase of the Kalman filter.

[0045] The state vector is defined as shown in Equation (1). u represents the horizontal coordinate of the center position of the object. v represents the vertical coordinate of the center position of the object. s represents the scale of the object (object width × object height). a represents the aspect ratio of the object (object width / object height). The dot denotes a time derivative. That is, dot u represents the amount of change (velocity) of the center position of the object in the horizontal direction. dot v represents the amount of change (velocity) of the center position of the object in the vertical direction. dot s represents the amount of change in the scale of the object.

[0046]

[0047] Subscripts t and k indicate that the state vector corresponds to the k-th tracking information in the t-th frame. In the following description, the subscript k is omitted, but the Kalman filter algorithm executed by the tracking unit 34 shall be executed for each piece of tracking information.

[0048] As described above, the tracking information may include information on the position, size, and velocity of the object. Among these, the position of the object corresponds to u and v in the state vector. The size of the object corresponds to s in the state vector. The velocity of the object corresponds to dot u and dot v in the state vector.

[0049] As shown in Equation (2), hat x, which is the a priori estimate of the state vector in the current frame t ｔ － can be derived using hat x, which is the a posteriori estimate of the state vector estimated in the immediately preceding frame (t-1) ｔ－１ F ｔ represents a state transition matrix. B ｔ represents a control input model. u ｔ represents an external control input to the Kalman filter.

[0050]

[0051] As shown in Equation (3), P, which is the a priori estimate of the error covariance matrix in the current frame t ｔ － can be derived using P, which is the a posteriori estimate of the error covariance matrix estimated in the immediately preceding frame (t-1) ｔ－１ Q ｔThis is the covariance matrix (process noise matrix) of the noise added to the model.

[0052]

[0053] In this embodiment, similar to the implementation of SORT, u ｔ Assume that u is zero, ｔ and B ｔ It is assumed that this does not affect the prediction of the state vector. Also, the state transition matrix F ｔ We define it as shown in equation (4), and model the motion of each object as following uniform linear motion.

[0054]

[0055] The prediction unit 35 predicts the tracking information for the current frame by deriving the state vector (see equation (2)) and the error covariance matrix (see equation (3)) in the Kalman filter algorithm based on the tracking information predicted, updated, or generated in past frames.

[0056] Furthermore, the prediction unit 35 updates the tracking degree parameters "hit_streak", "predict_flag", and "time_since_update" included in the tracking information. Specifically, if "time_since_update" is greater than 0, the prediction unit 35 clears "hit_streak" to "0". This is intended to reset the period during which object detection is ongoing, since the tracking information for the previous frame was not generated or updated based on Bbox. The prediction unit 35 also sets "predict_flag" to "True" and increments "time_since_update" by +1. This is intended to start counting the period during which only prediction is ongoing, since the tracking information based on Bbox has not been updated in the current frame.

[0057] The prediction unit 35 outputs tracking information, including the predicted state vector, error covariance matrix, and tracking degree parameter for the current frame, to the allocation unit 36. Hereafter, the predicted tracking information may be referred to as "predicted_tracker".

[0058] Furthermore, since Bbox represents the position and size of the object, it can be expressed using u, v, s, and a in the state vector. The width w and height h of Bbox can be obtained by the following transformation using s and a in the state vector: w = √(s × a) h = s / w

[0059] The allocation unit 36 ​​determines the predicted N ｐｒｅｄ The Bbox included in the individual tracking information and the detected N ｄｅｔｅｃｔ The Bboxes are associated with each other. Specifically, the assignment unit 36 ​​calculates the weight for each combination of Bboxes based on the degree of overlap between Bboxes using IoU (Intersection of Union), visual similarity, and / or spatial distance. The assignment unit 36 ​​then uses the weights for each combination as elements for N ｐｒｅｄ ×N ｄｅｔｅｃｔ A weight table is generated, and the Hungarian algorithm is applied to the weight table. That is, the allocation unit 36 ​​solves the correspondence between Bboxes as a maximum weight matching problem of a bipartite graph.

[0060] The allocation unit 36 ​​determines the predicted N ｐｒｅｄ For each Bbox included in the tracking information, the combination of the predicted tracking information and the detected Bbox that has been matched is output to the update unit 37. The update unit 37 uses this combination of predicted tracking information and the detected Bbox to perform the Kalman filter update process described later.

[0061] From now on, when Bboxes are matched with each other, this will be referred to as "successful tracking," and when Bboxes are not matched with each other, this will be referred to as "failed tracking." Additionally, successful tracking information may be called "matched_tracker," and unsuccessful tracking information may be called "unmatched_tracker."

[0062] The allocation unit 36 ​​determines the predicted N ｐｒｅｄFor Bboxes included in the tracking information that have not been matched with any detected Bboxes, the system determines whether or not to discard the predicted tracking information. Specifically, the allocation unit 36 ​​discards the predicted tracking information if it has not been updated by the Kalman filter for a certain period of time. For example, if the "time_since_update" included in the tracking information is greater than a predetermined period (hereinafter referred to as "max_age"), the allocation unit 36 ​​discards the tracking information and does not output it to the prediction unit 35, the update unit 37, or the control unit 40. On the other hand, if the "time_since_update" included in the tracking information is less than or equal to "max_age", the allocation unit 36 ​​does not discard the tracking information and outputs it to the prediction unit 35 and the control unit 40.

[0063] The allocation unit 36 ​​is the detected N ｄｅｔｅｃｔ For any Bbox that was not matched with a Bbox included in the predicted tracking information, new tracking information is generated based on the location and size indicated by the detected Bbox. The allocation unit 36 ​​also outputs the newly generated tracking information to the prediction unit 35 and the control unit 40. Hereafter, the newly generated tracking information may be referred to as "new_tracker".

[0064] Furthermore, the allocation unit 36 ​​sets the tracking degree parameters "hit_streak", "predict_flag", and "time_since_update" included in the newly generated tracking information. Specifically, the allocation unit 36 ​​sets "predict_flag" to "False", "time_since_update" to "0", and "hit_streak" to "0". This is intended to set the initial state for the newly generated tracking information.

[0065] The update unit 37 derives the true state vector (see equation (5)) and the error covariance matrix (see equation (6)) based on the tracking information (matched_tracker) output from the allocation unit 36 ​​and the detected Bbox associated with the tracking information. This updates the tracking information for the current frame.

[0066] As shown in equation (5), the true state vector is hat x ｋ This is the hat x, which is the prior estimate of the predicted state vector. ｋ － It can be derived using the following equation (6): As shown in equation (6), the true error covariance matrix P ｋ P is the prior estimate of the predicted error covariance matrix. ｋ － It can be derived using [this method].

[0067]

[0068] K ｋ H is the Kalman gain and is derived by equation (7). ｋ This represents a predetermined observation matrix. ｋ This represents the covariance matrix of noise included in the predetermined observed values ​​(observation noise matrix). I represents the identity matrix.

[0069]

[0070] z ｋ x is an observed value and can be converted using equation (8) with the information of Bbox (i.e., Bbox detected from the current frame by the detection unit 32) associated with the tracking information (matched_tracker) output from the allocation unit 36. ｃ y indicates the horizontal coordinate of the center of Bbox. ｃ The vertical coordinates of the center of Bbox are shown. w×h indicates the scale of Bbox. w / h indicates the aspect ratio of Bbox.

[0071]

[0072] Furthermore, the update unit 37 updates the tracking degree parameters "hit_streak", "predict_flag", and "time_since_update" included in the tracking information. Specifically, the update unit 37 sets "predict_flag" to "False", clears "time_since_update" to "0", and increments "hit_streak" by +1. This is intended to reset the period since the tracking information was last updated and increase the object detection continuation count, since the tracking information has been updated based on the Bbox detected in the current frame.

[0073] The update unit 37 outputs tracking information, including the updated state vector, error covariance matrix, and tracking degree parameters, to the prediction unit 35 and the control unit 40. Hereafter, the tracking information updated by the update unit 37 may be referred to as "updated_tracker".

[0074] The prediction unit 35 stores the tracking information output from the allocation unit 36 ​​(unmatched_tracker and new_tracker that were not discarded) and the tracking information output from the update unit 37 (updated_tracker) in the storage 14 or the like. The prediction unit 35 uses the stored tracking information for prediction processing in subsequent frames.

[0075] As a result of the above processing, the Bboxes detected by the detection unit 32 that were successfully tracked will be output to the control unit 40 with the updates from the update unit 37 reflected (included in updated_tracker). The Bboxes updated by the Kalman filter will have less variation in position and size between frames due to the smoothing performed by the Kalman filter. For example, when the movement of an object is relatively simple, such as when an object moves in a straight line, the smoothing performed by the Kalman filter is often advantageous because it reduces the noise included in the detection results.

[0076] On the other hand, there are cases where the Kalman filter's smoothing makes it difficult to track the actual movement of the object in the video, such as when an object moves irregularly. In such cases where the Kalman filter has difficulty tracking the actual movement of the object, it is common practice to automatically adjust the Kalman gain during state vector estimation so that the observed values ​​obtained in the current frame are reflected more in the estimation result than the predicted values ​​of the Kalman filter.

[0077] However, if, for example, the automatic adjustment of the Kalman gain cannot keep up with the movement of the object, ROI coding may be performed based on the Bbox information detected by the detection unit 32, without using the Bbox included in the tracking information after updating by the Kalman filter. Specifically, the update unit 37 calculates the updated error covariance matrix P ｋ Depending on the values ​​of variance and / or covariance included, the control unit 40 may be switched to output either the updated Bbox or the Bbox as detected. For example, if the values ​​of variance and / or covariance are greater than or equal to a predetermined threshold, the update unit 37 may replace the Bbox information included in updated_tracker with the Bbox information detected by the detection unit 32 and then output it to the control unit 40.

[0078] [Details of the Control Unit 40] Next, the detailed functional configuration of the control unit 40 will be described. The control unit 40 includes a determination unit 42, a decision unit 44, and a modification unit 46. As an example, the control unit 40 uses a quantization parameter (QP) as the coding parameter. Therefore, hereafter, the coding parameter will be referred to as "coding parameter QP".

[0079] The determination unit 42 determines whether each region within a frame is a region of interest based on the tracking information output by the object tracking unit 30. Specifically, the determination unit 42 determines, based on the tracking information, whether the object detected in each frame is included in the region of interest that includes the object to be processed.

[0080] For example, the determination unit 42 determines the image region enclosed by Bbox and the image region adjacent to Bbox included in the tracking information as the region of interest, and determines the other regions as the region of non-interest. The image region adjacent to Bbox refers to the region outside of Bbox that is included in the range that is separated from the boundary line of Bbox by a specified number of pixels. The specified number of pixels is, for example, a value that is set in advance by the user and stored in the storage 14. If the specified number of pixels is set to "0", the image region enclosed by Bbox itself becomes the region of interest.

[0081] Figure 5 shows an example of the positional relationship between the Bbox included in the tracking information, the area of ​​interest determined based on the Bbox, and each CTU within the frame. In Figure 5, an example is shown where the areas of interest based on each of the three Bboxes 1 to 3 overlap in a frame divided into multiple CTUs. Areas other than the areas of interest are non-areas of interest. In Figure 5, CTUs that do not overlap with any of the areas of interest are shaded, while CTUs that overlap with any of the areas of interest are not shaded, thereby distinguishing between CTUs that overlap with the areas of interest and those that do not. Note that overlap between a CTU and an area of ​​interest means that the area of ​​interest overlaps with the CTU even slightly.

[0082] The determination unit 44 determines the encoding parameters for each region within the frame based on the determination result by the judgment unit 42. When the encoding parameters are quantization parameters, setting the encoding parameter QP lower increases the amount of encoding but improves image quality. On the other hand, setting the encoding parameter QP higher decreases the amount of encoding but deteriorates image quality. For example, the determination unit 44 aims to improve the image quality of the region of interest by lowering the encoding parameter QP of the CTUs that overlap the region of interest, while reducing the amount of encoding of the entire image by raising the encoding parameter QP of the CTUs that do not overlap the region of interest.

[0083] Specifically, the determination unit 44 determines a reference encoding parameter QP for controlling image quality for each CTU unit within a frame, based on a target encoding amount predetermined by the user and the encoding amount obtained from the encoding unit 22. For example, since there is generally a strong correlation between the target encoding amount and the encoding parameter QP, the determination unit 44 may set the encoding parameter QP higher when the target encoding amount is small, and lower when the target encoding amount is large. Hereafter, the reference encoding parameter QP determined by the determination unit 44 will be referred to as "encoding parameter BaseQP".

[0084] The determination unit 44 determines the coding parameter BaseQP using a known method. The determination unit 44 may determine the coding parameter BaseQP using a value predetermined by the user, for example. Alternatively, the determination unit 44 may determine the coding parameter BaseQP by performing code amount control using the R-λ model on the target code amount assigned to the CTU, for example, as done in HEVC reference software (HM) and VVC reference software (VTM).

[0085] Incidentally, since the detection results of Bbox can change from frame to frame, it can be difficult to continuously and stably detect the same object. As a result, the determination of the region of interest and the region of non-interest based on Bbox also fluctuates from frame to frame. Therefore, if the encoding parameter QP is determined solely based on whether or not it is a region of interest, the encoding parameter QP will fluctuate significantly from frame to frame. This can potentially impair the subjective and objective image quality of the region of interest.

[0086] Therefore, the modification unit 46 modifies the coded parameter BaseQP based on the tracking information output by the object tracking unit 30 so that the coded parameter QP does not fluctuate significantly from frame to frame. Specifically, the modification unit 46 acquires at least one of the index scale related to the size of the object and the index score related to the confidence level of the object detection result, and modifies the coded parameter QP using at least one of the index scale and the index score.

[0087] More specifically, the modification unit 46 determines dQP, which is the modification amount of the encoding parameter BaseQP, on a CTU basis, and uses the encoding parameter BaseQP and the modification amount dQP to determine the final encoding parameter QP to be applied to the CTU for each CTU. As an example, the modification amount dQP is an integer less than or equal to 0, and the encoding parameter QP to be applied to the CTU is calculated by the following formula: QP = BaseQP + dQP

[0088] The method for modifying the coding parameter QP is not limited to the method described above. For example, if the modification amount dQP is a weight coefficient represented by a real number between 0 and 1, the coding parameter QP may be calculated using the following formula: QP = BaseQP × dQP

[0089] An example of how to calculate the correction amount dQP is described below. First, the correction unit 46 obtains an index called scale related to the size of the object. For example, the correction unit 46 calculates the index scale representing the size of the object from Bbox included in the tracking information using the following formula: w represents the width of Bbox. h represents the height of Bbox. MaxScale is a predetermined parameter for normalizing scale to a real value between 0 and 1. scale = max(w × h, MaxScale) / MaxScale

[0090] Next, the correction unit 46 acquires a score, which is an index related to the reliability of the object detection result. For example, the correction unit 46 acquires the reliability score attached to Bbox output from the object detector as the score. The correction unit 46 may also calculate the score using an arbitrary calculation formula that uses the reliability score attached to Bbox.

[0091] Next, the correction unit 46 calculates a weight coefficient r for correcting the coding parameter QP using the index score and the calculated index scale, according to the following formula: The weight coefficient r is the weighted average value of scale and score. r = α × scale + (1 - α) × score

[0092] α is a predetermined parameter, and its value is, for example, a real number between 0.0 and 1.0. In the above formula, the larger α is, the more the object size index scale is reflected in the weight coefficient r than the confidence index score. Note that if the value of "predict_flag" in the tracking information is "True", the tracking information does not reflect the information of the Bbox detected in the current frame. Therefore, in this case, by setting α to 1.0, the weight coefficient r can be determined without using the value of the index score.

[0093] The correction unit 46 calculates the correction amount dQP using the index scale and the weight coefficient r calculated using the index score. In this case, the method of calculating the correction amount dQP is changed according to the value of "predict_flag" included in the tracking information. If the value of "predict_flag" is "False", it means that the tracking information was successful in tracking an object or that it was newly generated based on a Bbox detected in the current frame. Therefore, for this tracking information, the correction unit 46 controls the correction amount dQP to change in the negative direction in order to improve image quality by setting the encoding parameter QP of the corresponding area of ​​interest to a lower value.

[0094] On the other hand, if the value of "predict_flag" is "True", it means that the tracking information failed. That is, it means that detection or tracking failed even though an object actually exists, or that the object actually disappeared from the image. Therefore, the correction unit 46 controls the correction amount dQP to change in the positive direction in order to set the encoding parameter QP of the corresponding area of ​​interest high for this tracking information and degrade the image quality. The following describes specific examples of how to calculate the correction amount dQP for each case.

[0095] First, let's explain the case where the value of "predict_flag" is "False". The correction unit 46 calculates the change in the correction amount dQP, ΔdQP, using the following formula. The subscripts t and k indicate that ΔdQP corresponds to the k-th tracking information in the t-th frame. The subscripts t and k are also added to the weight coefficient r so that it can be distinguished according to the frame and tracking information. ΔdQPｔｒａｃｋｅｄ ΔdQP is a predetermined parameter that takes a value less than 0. ｔ，ｋ = r ｔ，ｋ ×ΔdQP ｔｒａｃｋｅｄ

[0096] Weight coefficient r ｔ，ｋ The value of increases as the confidence score of Bbox increases and as Bbox increases. In other words, according to the above formula, the change amount ΔdQP decreases significantly as the confidence score of Bbox included in the tracking information increases and as Bbox increases, thereby aiming to immediately improve image quality.

[0097] Then, the correction unit 46 calculates the change amount ΔdQP ｔ，ｋ Using this, the correction amount dQP for the coding parameter QP ｔ，ｋ Calculate the correction amount dQP. ｔ，ｋ For example, one of the methods in Examples 1 to 3 below can be applied to calculate this.

[0098] (Example 1) The modification unit 46 modifies the coding parameter QP set in the previous frame by a modification amount dQP ｔ－１，ｋ The calculated change ΔdQP ｔ，ｋ By adding this, the correction amount dQP of the encoding parameter QP in the current frame is obtained. ｔ，ｋ You may also calculate it. In this case, the calculation formula is expressed as follows: dQP ｔ，ｋ = dQP ｔ－１，ｋ +ΔdQP ｔ，ｋ

[0099] (Example 2) The modification unit 46 changes the value of "hit_streak[k]" corresponding to the k-th tracking information by a change amount ΔdQP ｔ，ｋ By multiplying by this, the correction amount dQP of the encoding parameter QP in the current frame is obtained. ｔ，ｋ You may also calculate the following: "hit_streak[k]" is a counter that indicates how many frames the tracking has been successful for the k-th tracking information. In other words, the more successful the tracking continues, the greater the correction amount dQP of the encoding parameter QP. ｔ，ｋ The degree of change increases. The calculation formula in this case is expressed as follows: dQP ｔ，ｋ = dQPｔ－１，ｋ +ΔdQP ｔ，ｋ ×hit_streak[k]

[0100] (Example 3) The modification unit 46 uses an arbitrary function FΔdqp with "hit_streak[k]" as the variable to determine the change amount ΔdQP ｔ，ｋ The term relating to this may be determined. In this case, the calculation formula is expressed as follows: dQP ｔ，ｋ = dQP ｔ－１，ｋ +FΔdqp(hit_streak[k])

[0101] Figure 6 is a graph showing an example of a function FΔdqp for deriving the change ΔdQP with "hit_streak" as the variable. In Figure 6, the function FΔdqp is set such that the change ΔdQP increases in the negative direction as "hit_streak" increases. Such a function can be realized, for example, by a sigmoid function as shown in the following equation: FΔdqp(x) = -1 / (1 + exp(-a × (x - x0))) + Offset

[0102] Alternatively, the weight coefficient r may be set to the slope of the sigmoid function. That is, the higher the confidence score of the object and the larger Bbox is, the greater the reduction in the correction amount dQP, even with a small change in "hit_streak". In this case, the sigmoid function is expressed by the following formula: FΔdqp(x) = -1 / (1 + exp(-r × (x - x0))) + Offset

[0103] To avoid performing calculations with real numbers each time, the values ​​of FΔdqp corresponding to "hit_streak" may be pre-created as a table and stored in storage 14 or similar.

[0104] Next, we will explain the case where the value of "predict_flag" is "True". The correction unit 46 calculates the change amount ΔdQP of the correction amount dQP using the following formula: ΔdQP ｕｎｒａｃｋｅｄ ΔdQP is a predetermined parameter that takes a value greater than 0. ｔ，ｋ = (1 - r ｔ，ｋ ) × ΔdQP ｕｎｒａｃｋｅｄ

[0105] Weight coefficient rｔ，ｋ The coefficient (1-r) used ｔ，ｋ The value of ) increases as the confidence score of Bbox decreases and Bbox becomes smaller. In other words, according to the above formula, the lower the confidence score of Bbox included in the tracking information and the smaller Bbox becomes, the greater the change in ΔdQP, thereby immediately degrading the image quality.

[0106] Then, the correction unit 46 calculates the change amount ΔdQP ｔ，ｋ Using this, the correction amount dQP for the coding parameter QP ｔ，ｋ Calculate the correction amount dQP. ｔ，ｋ For example, one of the methods in Examples 4 to 6 below can be applied to calculate this.

[0107] (Example 4) The modification unit 46 modifies the coding parameter QP set in the previous frame by a modification amount dQP ｔ－１，ｋ The calculated change ΔdQP ｔ，ｋ By adding this, the correction amount dQP of the encoding parameter QP in the current frame is obtained. ｔ，ｋ You may also calculate it. In this case, the calculation formula is expressed as follows: dQP ｔ，ｋ = dQP ｔ－１，ｋ +ΔdQP ｔ，ｋ

[0108] (Example 5) The modification unit 46 changes the value of "time_since_update[k]" corresponding to the k-th tracking information by a change amount ΔdQP ｔ，ｋ By multiplying by this, the correction amount dQP of the encoding parameter QP in the current frame is obtained. ｔ，ｋ You may also calculate the following: "time_since_update[k]" is a counter that indicates how many frames the k-th tracking information has been continuously predicted. In other words, the more continuously the tracking is predicted, the greater the correction amount dQP of the coding parameter QP. ｔ，ｋ The degree of change increases. The calculation formula in this case is expressed as follows: dQP ｔ，ｋ = dQP ｔ－１，ｋ +ΔdQP ｔ，ｋ ×time_since_update[k]

[0109] (Example 6) The modification unit 46 uses an arbitrary function FΔdqp with "time_since_update[k]" as a variable to determine the change amount ΔdQP ｔ，ｋ The term relating to this may be determined. In this case, the calculation formula is expressed as follows: dQP ｔ，ｋ = dQP ｔ－１，ｋ +FΔdqp(time_since_update[k])

[0110] Figure 7 is a graph showing an example of a function FΔdqp for deriving the change ΔdQP with "time_since_update" as the variable. In Figure 7, the function FΔdqp is set such that the change ΔdQP increases in the positive direction as "time_since_update" increases. Such a function can be realized, for example, by a sigmoid function as shown in the following equation: FΔdqp(x) = 1 / (1 + exp(-a × (x - x0))) + Offset

[0111] Alternatively, a coefficient (1-r) using a weight coefficient r may be set as the slope of the sigmoid function. That is, the lower the confidence score of the object and the smaller Bbox is, the greater the reduction in the correction amount dQP, even with small changes in "time_since_update". In this case, the sigmoid function is expressed by the following formula: FΔdqp(x) = 1 / (1 + exp(-(1-r) × (x-x0))) + Offset

[0112] To avoid performing calculations with real numbers each time, the values ​​of FΔdqp corresponding to "time_since_update" may be pre-created as a table and stored in storage 14 or similar.

[0113] Furthermore, the modification unit 46 modifies the coding parameter QP by a modification amount dQP. ｔ，ｋ It is preferable to store the value in storage 14, etc., each time it is calculated. This allows the correction amount dQP to be corrected by the methods of Examples 1 to 6. ｔ，ｋ When calculating this, the correction amount dQP of the coding parameter QP set in the previous frame is used. ｔ－１，ｋ This allows for reference and improved efficiency in calculations.

[0114] FIG. 8 shows the correction amount dQP of the encoding parameter QP ｔ，ｋ shows an example of the configuration of a table for holding. The ID is identification information of an object and is included in tracking information. The correcting unit 46 holds the correction amount dQP of the encoding parameter QP in association with the ID.

[0115] After calculating the correction amount dQP of the encoding parameter QP ｔ，ｋ , the correcting unit 46 may convert the correction amount dQP ｔ，ｋ into an integer by rounding off or the like, and perform clipping processing so that the result falls within the range of a predetermined target encoding parameter correction amount dQP ｔａｒｇｅｔ . The calculation formula in this case is represented by the following formula. Int is a function that rounds to an integer by rounding off. Clip is a function for bringing a first argument into a range not less than a second argument and not more than a third argument. According to the following formula, the integerized correction amount dQP ｔ，ｋ is contained in a range that is not less than the target encoding parameter correction amount dQP ｔａｒｇｅｔ and not more than 0. dQP ｔ，ｋ = Clip(Int(dQP ｔ，ｋ ), dQP ｔａｒｇｅｔ , 0)

[0116] FIG. 9 shows an example of a change for each frame (time point) of the correction amount dQP of the encoding parameter QP calculated for a specific object. It is shown that at time points t0 to t3 and t5 to t6, tracking of the object succeeds, and the correction amount dQP of the encoding parameter QP decreases or maintains the same value as before. It is shown that at time point t4 and t7 to t11, tracking of the object fails, and the correction amount dQP of the encoding parameter QP increases. As described above, by using tracking information, abrupt fluctuation of dQP between frames according to an object detection result and abrupt image quality change accompanying the fluctuation of dQP can be suppressed.

[0117] Note that, when the correction amount dQP ｔ，ｋ is not "0" and the corresponding k-th tracking information is discarded by the tracking unit 34, the correcting unit 46 ｔ，ｋ initializes the correction amount dQP to 0.

[0118] Furthermore, in a case where a plurality of different correction amounts dQP are set, such as a case where a plurality of different Bboxes overlap for the same CTU, the correction unit 46 may determine the final correction amount dQP using at least some of the plurality of different correction amounts dQP based on a predetermined rule. For example, the correction unit 46 may use the correction amount dQP obtained in the immediately preceding frame ｔ－１ to select the correction amount dQP whose value is closest ｔ Alternatively, for example, the correction unit 46 may use a plurality of different correction amounts dQP ｔ representative values such as an average value, a median value, a maximum value, and a minimum value of may be used.

[0119] Furthermore, the method for calculating the correction amount dQP of the encoding parameter QP is not limited to the method using the index scale related to the size of an object and the index score related to the reliability of an object detection result as described above. For example, the correction unit 46 may use either one of the index scale and the index score. Further, for example, the correction unit 46 may calculate the correction amount dQP using various parameters such as the speed of the object, the variance of the position of the object, and the variance of the size of the object.

[0120] Furthermore, the method for correcting the encoding parameter QP using tracking degree parameters is not limited to the method described above. For example, the correction unit 46 may correct the encoding parameter QP using some of "hit_streak", "predict_flag", and "time_since_update". Further, for example, any other tracking degree parameter may be used, such as the cumulative number of successful tracking of an object and the cumulative number of failed tracking from when tracking information is generated to when it is discarded.

[0121] Next, the operation of the image encoding apparatus 10 will be described. FIG. 10 is a flowchart showing the flow of image encoding processing performed by the image encoding apparatus 10. The image encoding processing is performed when the CPU 11 reads an image encoding program from the ROM 12 or the storage 14, develops it in the RAM 13, and executes the program. The image encoding processing is an example of the image encoding method of the present disclosure.

[0122] In step S10, the CPU 11, acting as the preprocessing unit 20, acquires the video input from the camera 18. In step S12, the CPU 11, acting as the preprocessing unit 20, performs preprocessing on the acquired video. Specifically, the CPU 11, acting as the preprocessing unit 20, converts the video into a format that can be processed by the encoding unit 22 and the object tracking unit 30, and stores the converted video in the storage 14.

[0123] In step S14, the CPU 11, acting as a detection unit 32, detects specific objects within each frame that make up the acquired video and outputs a Bbox as the detection result. In step S16, the CPU 11, acting as a tracking unit 34, executes a tracking process and outputs tracking information for each object. The flow of the tracking process will be described later.

[0124] In step S18, the CPU 11, as a determination unit 42, determines whether each region within the frame (e.g., CTU) is a region of interest based on the tracking information. Specifically, the CPU 11, as a determination unit 42, determines that the image region enclosed by Bbox included in the tracking information, and the image region adjacent to Bbox, are regions of interest, and determines that the other regions are not regions of interest. In step S20, the CPU 11, as a decision unit 44, determines the encoding parameter BaseQP for each CTU (region within the frame) based on the determination result of the region of interest.

[0125] In step S22, the CPU 11, as a modification unit 46, calculates the encoding parameter QP for each CTU by modifying the encoding parameter BaseQP for each CTU based on the tracking information. In step S24, the CPU 11, as an encoding unit 22, encodes each CTU using the encoding parameter QP for each CTU. Once step S24 is completed, the CPU 11 terminates the image encoding process.

[0126] Figure 11 is a flowchart showing an example of the tracking process flow by the tracking unit 34 (prediction unit 35, allocation unit 36, and update unit 37). The tracking process is performed in step S16 of the image encoding process.

[0127] In step S50, the CPU 11, as the prediction unit 35, predicts the tracking information for the current frame using a Kalman filter based on the tracking information predicted, updated, or generated in past frames. In step S52, the CPU 11, as the prediction unit 35, refers to "time_since_update" included in the predicted tracking information, and if its value is greater than "0", proceeds to step S54. In step S54, the CPU 11, as the prediction unit 35, clears "hit_streak" included in the predicted tracking information to "0".

[0128] On the other hand, if step S52 is denied (i.e., "time_since_update" is "0"), the process proceeds to step S56. In step S56, the CPU 11, as the prediction unit 35, sets "predict_flag" included in the predicted tracking information to "True" and increments "time_since_update" by +1.

[0129] In step S58, the CPU 11, acting as the allocation unit 36, acquires the Bbox detected in step S14. In step S60, the CPU 11, acting as the allocation unit 36, associates the Bbox included in the predicted tracking information with the detected Bbox. In step S62, the CPU 11, acting as the allocation unit 36, determines whether the association in step S60 was performed (i.e., whether the tracking was successful). If the association is successful, the process proceeds to step S64.

[0130] In step S64, the CPU 11, acting as the update unit 37, updates the successfully associated tracking information using a Kalman filter based on the detected Bbox associated with the tracking information. The CPU 11 also clears "time_since_update" to "0", sets "predict_flag" to "False", and increments "hit_streak" by +1.

[0131] On the other hand, if step S62 is denied (if the mapping in step S60 is not performed and tracking fails), the process proceeds to step S66. In step S66, the CPU 11, as the allocation unit 36, refers to "time_since_update" included in the predicted tracking information, and if its value is greater than "max_age", the process proceeds to step S68. In step S68, the CPU 11, as the allocation unit 36, discards the predicted tracking information. If step S68 is completed, or if step S66 is denied (if "time_since_update" is less than or equal to "max_age"), the process proceeds to step S70.

[0132] In step S70, the CPU 11, as the allocation unit 36, generates new tracking information for the Bboxes acquired in step S58 that were not matched in step S60. The CPU 11 also sets "time_since_update" to "0", "predict_flag" to "False", and "hit_streak" to "0" as the allocation unit 36.

[0133] In step S72, the CPU 11, as a prediction unit 35, holds the tracking information predicted in step S50 that was not updated in step S64 or discarded in step S68, the tracking information updated in step S64, and the tracking information generated in step S70. The CPU 11, as a tracking unit 34, also outputs each of these pieces of tracking information to the control unit 40. Once step S72 is completed, the CPU 11 terminates the tracking process.

[0134] In this manner, the tracking unit 34 acquires the Bbox detected for each frame, predicts tracking information using a Kalman filter, associates Bboxes with each other, and updates the tracking information using a Kalman filter.

[0135] As described above, the image encoding device 10 according to this embodiment includes an object tracking unit 30 that tracks objects in each frame constituting the image and outputs tracking information including the degree to which object tracking has been successful or unsuccessful from the past to the present; a determination unit 42 that determines whether each region within the frame is a region of interest based on the tracking information; a determination unit 44 that determines encoding parameters for controlling image quality for each region within the frame based on the determination result; and a modification unit 46 that modifies the encoding parameters based on the tracking information.

[0136] According to the image encoding device 10, even if, for example, the detection result of Bbox changes from frame to frame, the encoding parameter QP can be appropriately modified based on tracking information so that it does not fluctuate significantly from frame to frame. Therefore, frame-to-frame changes in image quality can be suppressed. This makes it possible to suppress the deterioration of subjective and objective image quality, especially in the area of ​​interest.

[0137] [Second Embodiment] In the image encoding device 10 according to the first embodiment, if tracking of an irregularly moving object fails, the predicted tracking information (predicted_tracker) is expected to be inaccurate, as it deviates from the actual object's position. In this case, the image quality of the area that is inaccurate may be improved, potentially unnecessarily increasing the bitrate. On the other hand, if the object's movement is regular, even if tracking fails, the predicted tracking information is likely to accurately predict the actual object's position. In this case, it is considered reasonable to maintain improved image quality in the area corresponding to the object's position indicated by the predicted tracking information.

[0138] Therefore, in the second embodiment, the change amount ΔdQP of the correction amount dQP of the coding parameter QP calculated by the correction unit 46 is set based on the movement of the object and its variability, in addition to the index scale related to the size of the object and the index score related to the reliability of the object detection result. Note that explanations that overlap with the first embodiment are omitted.

[0139] Specifically, the correction unit 46 derives at least one of a first index relating to the motion of the object and a second index relating to the variability of the motion of the object, based on the tracking information output by the object tracking unit 30. For example, the correction unit 46 derives the hat x, which is the state vector of the Kalman filter included in the tracking information. ｋ Based on (see equation (5)), the magnitude of the object's motion mv ｔ，ｋ The magnitude of motion mv may also be calculated using equation (9). ｔ，ｋ This is an example of the first indicator.

[0140]

[0141] For example, the correction unit 46 adjusts the error covariance matrix P of the Kalman filter included in the tracking information. ｋ Based on (see equation (6)), the index σ represents the variability of an object's motion. ｔ，ｋ σ may also be calculated using formula (10). ｕ ２ σ represents the dispersion of the horizontal velocity at the center of the object. ｖ ２ This indicates the dispersion of the velocity perpendicular to the center of the object. The index σ ｔ，ｋ This is an example of a second indicator.

[0142]

[0143] The modification unit 46 modifies the coding parameter using at least one of the first and second indicators. As an example, we will describe an example in which the value of "predict_flag" included in the tracking information is "True," and the modification amount dQP of the coding parameter QP is calculated using the method of Example 6. In this case, the formula for calculating the modification amount dQP is expressed as follows: dQP ｔ，ｋ = dQP ｔ－１，ｋ +FΔdqp(time_since_update[k])

[0144] The function FΔdqp can be realized by setting the coefficient β×(1-r), which uses weight coefficients r and β, as the slope of the sigmoid function. In this case, the sigmoid function is expressed by the following equation: FΔdqp(x) = 1 / (1 + exp(-β×(1-r)×(x-x0))) + Offset

[0145] The coefficient β is, for example, the magnitude of the object's motion mv in the previous frame. ｔ－１，ｋ and σ, an index of the variability of object motion in the previous frame. ｔ－１，ｋ It can be set based on the magnitude of the object's motion mv. ｔ－１，ｋ The predetermined threshold Th ｍｖ That is all, and the index σ related to the variability of object motion. ｔ－１，ｋ The predetermined threshold Th σ If the following conditions are met, β is set to a predetermined real value between 0 and 1. For example, the modification unit 46 considers the magnitude of the object's motion mv. ｔ－１，ｋ The predetermined threshold Th ｍｖ It is less than σ, and is an index related to the variability of the object's motion. ｔ－１，ｋ The predetermined threshold Th σ If it is greater than β, set β to 1.

[0146] By setting the coefficient β in this way, if object tracking fails, the slope of the sigmoid function is set to be small for objects whose movement in past frames was large and whose movement variability was small, and the correction amount dQP of the coding parameter QP is adjusted. ｔ，ｋ The amount of increase is reduced. Therefore, for objects whose movement is large and regular and whose movement is easy to predict, the image quality of the region corresponding to the object can be maintained while being improved compared to the first embodiment. Furthermore, by setting the coefficient β, for objects whose movement is small and irregular and whose movement is difficult to predict, the image quality of the region corresponding to the object can be immediately reduced compared to the first embodiment.

[0147] Furthermore, each time the correction unit 46 calculates the correction amount dQP, it calculates the value of dQP and the magnitude of the motion of the object used in the calculation mv ｔ，ｋ and an index related to the variability of object motion σ ｔ，ｋIt is preferable to associate these with each other and store them in storage 14, etc. Figure 12 shows an example of the configuration of a table for storing the correction amount dQP. The ID is the identification information of the object and is included in the tracking information. This makes it possible to refer to each value that has already been calculated in the previous frame when calculating the correction amount dQP and setting the coefficient β, thereby improving the efficiency of processing.

[0148] Furthermore, the calculation of the correction amount dQP may be performed using only one of the first or second indicators. When only the first indicator is used, the image quality of the corresponding region can be improved while maintaining its quality for objects with large motion. When only the second indicator is used, the image quality of the corresponding region can be improved while maintaining its quality for objects with regular motion.

[0149] Also, the error covariance matrix P ｋ This includes variance and covariance related to the object's position, size, and aspect ratio, etc. The modification unit 46 may also use these variables as a second indicator.

[0150] [Third Embodiment] In the third embodiment, the conditions used for determining discard are changed so that tracking information is less likely to be discarded for objects whose movement is large, regular, and easily predictable. This makes it easier to maintain improved image quality in the area corresponding to the object, even if tracking fails repeatedly.

[0151] Specifically, the object tracking unit 30 derives at least one of a first index relating to the movement of the object and a second index relating to the variability of the movement of the object, based on the tracking information output by the tracking unit 34. For example, the object tracking unit 30 derives the magnitude of the movement of the object mv from the tracking information using a method similar to that of the modification unit 46 in the second embodiment. ｔ，ｋ and an index related to the variability of object motion σ ｔ，ｋ At least one of the following is calculated and stored in storage 14, etc. Alternatively, the object tracking unit 30 calculates the magnitude of motion mv calculated by the correction unit 46. ｔ，ｋ and indicator σ ｔ，ｋ You may obtain it.

[0152] The object tracking unit 30 changes the conditions used to determine whether or not to discard tracking information using at least one of the first and second indicators. Specifically, first, the object tracking unit 30 determines whether the tracking information that failed to track (i.e., tracking information that may be discarded) pertains to an object with large movement and little variation in movement. For example, the object tracking unit 30 determines the magnitude of the object's movement mv in the previous frame. ｔ－１，ｋ The predetermined threshold Th ｍｖ That is all, and the index σ, which represents the variability of object motion in the previous frame. ｔ－１，ｋ The predetermined threshold Th σ An object is determined to have large movement and low variation in movement if the following conditions are met:

[0153] If the object tracking unit 30 determines that the object is moving significantly and with little variation in its movement, it increments "max_age," which is used by the allocation unit 36 ​​to determine whether or not to discard the tracking information. For example, the object tracking unit 30 increments "max_age" by +1. The object tracking unit 30 also sets the value of "max_age_incr," a flag used to remember that "max_age" has been incremented, to "True." If the value of "max_age_incr" is "True" in subsequent frames, the object tracking unit 30 does not further increment "max_age." Furthermore, for tracking information that has been successfully tracked, the object tracking unit 30 initializes "max_age_incr" to "False" if the value is "True."

[0154] By controlling "max_age" in this way, tracking information for objects whose movements are large, regular, and easily predictable becomes less likely to be discarded than in the first and second embodiments. Therefore, the image quality of the region corresponding to the object can be maintained while improving.

[0155] Furthermore, the tracking unit 34 may decrement "max_age" if it determines that the object is moving slowly and irregularly. In this case, the tracking information is more likely to be discarded immediately than in the first and second embodiments.

[0156] Furthermore, the determination of whether or not to increment "max_age" may be made using only one of the first or second indicators. When only the first indicator is used, the image quality of the corresponding region can be improved for objects with large movements. When only the second indicator is used, the image quality of the corresponding region can be improved for objects with regular movements.

[0157] In the embodiments described above, the image encoding device 10 processes video captured by the camera 18. However, the video processed by the image encoding device 10 is not limited to video captured by the camera 18. For example, in the edge terminal 4, moving images such as animations composed of multiple images drawn by the user may be processed by the image encoding device 10.

[0158] In the embodiments described above, the detection unit 32 performs object detection processing to output the object detection result, but the invention is not limited to this. For example, the detection unit 32 may perform segmentation of a specific object and generate an importance map. When the detection unit 32 does not directly output the position, size, and confidence score of a specific object, such as when it generates an importance map, the position, size, and confidence score of a specific object may be determined by performing post-processing on the importance map. For example, the detection unit 32 may use the position and number of pixels in a group of pixels where the importance of each pixel shown in the importance map is above a predetermined threshold as the position and size of a specific object. Alternatively, the detection unit 32 may use the average value of the importance shown by each pixel included in the pixel group as the confidence score.

[0159] In the embodiments described above, the tracking unit 34 generates tracking information by executing an object tracking method using the object detection results such as SORT, but it is not limited to this. Any method capable of generating the above-described tracking information can be appropriately applied as the object tracking method executed by the tracking unit 34. For example, the tracking unit 34 may execute a classical object tracking method using the color, brightness, complexity, local features such as SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) of the original image, and / or motion vectors such as optical flow.

[0160] In the embodiments described above, the tracking unit 34 uses a Kalman filter to predict and update tracking information, but the invention is not limited to these embodiments. For example, when generating tracking information using motion vectors, motion vectors can also be used to predict the tracking information. In this case, the average value of multiple motion vectors contained in the image region surrounded by Bbox detected in past frames may be used as the velocity of the object. Alternatively, the position obtained by translating the past Bbox using the average value of the motion vectors may be used as the position of the predicted Bbox. Furthermore, the size and aspect ratio of the predicted Bbox may be the same as those of the past Bbox.

[0161] In the embodiments described above, a configuration in which quantization parameters are used as encoding parameters determined by the control unit 40 has been described, but the invention is not limited to this. For example, the target code amount may be used directly as the encoding parameter. In this case, the lower the encoding parameter is set, the lower the image quality, and the higher the encoding parameter is set, the higher the image quality. For example, the determination unit 44 may improve the image quality of the region of interest by setting a high encoding parameter that serves as the reference for CTUs that overlap the region of interest, while reducing the code amount of the entire image by setting a low encoding parameter that serves as the reference for CTUs that do not overlap the region of interest.

[0162] Furthermore, the image encoding process that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to execute particular processes, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit configurations can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits). The learning process and / or analysis process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0163] Furthermore, although the above embodiment describes an embodiment in which the image encoding program is pre-stored (installed) in ROM 12 or storage 14, the invention is not limited to this. The program may be provided in a form stored on a non-transitor storage medium such as CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), and USB (Universal Serial Bus) memory. Alternatively, the program may be provided in a form downloaded from an external device via a network.

[0164] The technology of this disclosure can also be appropriately combined with the above-described embodiments and modifications. The descriptions and illustrations shown above are detailed explanations of the parts relating to the technology of this disclosure and are merely examples of the technology of this disclosure. For example, the above descriptions of the configuration, function, operation, and effect are examples of the configuration, function, operation, and effect of the parts relating to the technology of this disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements added, or replaced from the descriptions and illustrations shown above, as long as they do not deviate from the spirit of the technology of this disclosure.

[0165] The following additional information is disclosed regarding the embodiments described above.

[0166] (Note 1) An image encoding device comprising: a memory; and at least one processor connected to the memory, wherein the processor tracks objects in each frame constituting an image, outputs tracking information including the degree to which tracking of the objects has been successful or unsuccessful from the past to the present; determines whether each region in the frame is a region of interest based on the tracking information; determines encoding parameters for controlling image quality for each region in the frame based on the result of the determination; modifies the encoding parameters based on the tracking information; and encodes each region in the frame using the encoding parameters.

[0167] (Appendix 2) A non-temporary storage medium storing a program executable for a computer to perform image encoding processing, wherein the image encoding processing includes: tracking objects in each frame constituting an image, outputting tracking information including the degree to which the tracking of the objects has been successful or unsuccessful from the past to the present; determining whether each region in the frame is a region of interest based on the tracking information; determining encoding parameters for controlling image quality for each region in the frame based on the result of the determination; modifying the encoding parameters based on the tracking information; and encoding each region in the frame using the encoding parameters.

[0168] (Note 3) An image encoding device comprising: an object tracking unit that tracks an object in each frame constituting an image and outputs tracking information including the degree to which the tracking of the object has been successful or unsuccessful from the past to the present; a determination unit that determines whether each region in the frame is a region of interest based on the tracking information; a determination unit that determines encoding parameters for controlling the image quality for each region in the frame based on the result of the determination; a modification unit that modifies the encoding parameters based on the tracking information; and an encoding unit that encodes each region in the frame using the encoding parameters.

[0169] (Appendix 4) The image encoding apparatus according to Appendix 3, wherein the modification unit acquires at least one of an index scale relating to the size of the object and an index score relating to the reliability of the detection result of the object, and modifies the encoding parameters based on the index scale and at least one of the index score, and the tracking information.

[0170] (Appendix 5) The image encoding apparatus according to Appendix 4, wherein the modification unit calculates the amount of change in the modification amount of the encoding parameter based on at least one of the index scale and the index score.

[0171] (Appendix 6) The image encoding apparatus according to Appendix 5, wherein the modification unit changes the method for calculating the modification amount of the encoding parameter using the change amount, depending on the degree to which the tracking of the object included in the tracking information was successful or unsuccessful.

[0172] (Appendix 7) The image encoding apparatus according to any one of Appendix 4 to Appendix 6, wherein the tracking information includes at least one of information indicating the size of the object and information indicating the reliability of the detection result of the object, and the modification unit acquires at least one of the index scale and the index score based on the tracking information.

[0173] 1 Event-driven inference system 2 Collaboration server 3 Cloud 4 Edge terminal 10 Image encoding device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input unit 16 Display unit 17 Communication interface 18 Camera 19 Bus 20 Preprocessing unit 22 Encoding unit 30 Object tracking unit 32 Detection unit 34 Tracking unit 35 Prediction unit 36 ​​Assignment unit 37 Update unit 40 Control unit 42 Judgment unit 44 Decision unit 46 Correction unit

Claims

1. An image encoding device comprising: an object tracking unit that tracks objects within each frame constituting an image and outputs tracking information including the degree to which the tracking of the objects has been successful or unsuccessful from the past to the present; a determination unit that determines whether each region within the frame is a region of interest based on the tracking information; a determination unit that determines encoding parameters for controlling image quality for each region within the frame based on the result of the determination; a modification unit that modifies the encoding parameters based on the tracking information; and an encoding unit that encodes each region within the frame using the encoding parameters.

2. The image encoding apparatus according to claim 1, wherein the modification unit derives at least one of a first index relating to the movement of the object and a second index relating to the variability of the movement of the object based on the tracking information, and modifies the encoding parameters using at least one of the first index and the second index.

3. The image encoding apparatus according to claim 1, wherein the object tracking unit derives at least one of a first index relating to the movement of the object and a second index relating to the variability of the movement of the object based on the tracking information, and modifies the conditions used to determine whether or not to discard the tracking information using at least one of the first index and the second index.

4. An image encoding method performed by a computer, comprising: tracking objects within each frame constituting an image; outputting tracking information including the degree to which tracking of the objects has been successful or unsuccessful from the past to the present; determining whether each region within the frame is a region of interest based on the tracking information; determining encoding parameters for controlling image quality for each region within the frame based on the result of the determination; modifying the encoding parameters based on the tracking information; and encoding each region within the frame using the encoding parameters.

Images