Training device, image processing device, training method, and training program

Abstract

This training device comprises: a region correction unit (12) that corrects a first bounding box indicating an object region where a target object detected from at least one camera image is located so that the first bounding box is suitable for identifying the target object, thereby generating a corrected bounding box indicating a corrected object region; and a training unit (13) that trains a learning model for image recognition using position information of the corrected bounding box.

Description

Learning device, image processing device, learning method, and learning program

[0001] The present disclosure relates to image recognition technology, and particularly relates to technology such as object search technology that requires object position information during learning.

[0002] In an object search technology that identifies the position of a specific object such as a person or a vehicle in a camera image using a trained model, a camera image, the region of the target object, and an ID label are required as training data for the model. However, annotating the region and ID label of the target object on the camera image requires enormous labor.

[0003] As a technology for reducing such labor, there is the technology disclosed in Non-Patent Document 1. Non-Patent Document 1 proposes a weakly supervised person search technology that requires annotation of bounding boxes on camera images, but for ID labels, clusters image feature values obtained from camera images to generate pseudo labels, and performs training of a learning model using the pseudo labels. According to this technology, since manual annotation of ID labels is not required, the labor for labeling can be reduced.

[0004] Weakly Supervised Person Search with Region Siamese Networks, ICCV, 2021.

[0005] Even with the technology taught in Non-Patent Document 1, the annotation of bounding boxes indicating object regions for the camera images of training data is still required, so the problem of requiring labor has not yet been solved.

[0006] In this regard, a method is also conceivable in which the result of object detection by an object detection model trained on a large-scale dataset is used as the object region. However, the results of object detection by such a model may lack information useful for object search, such as shoes, lower body, hands, arms, and belongings, and there is a problem that the object detection results by conventional models are not suitable for object search tasks.

[0007] This disclosure was made in response to the recognition of the above-mentioned problems, and aims to provide a technology that can reduce the effort required for annotating bounding boxes that indicate appropriate object regions for image recognition, including object search.

[0008] One aspect of the learning device according to the embodiment of the present disclosure includes a region correction unit that corrects a first bounding box indicating the object region where a target object is located, detected from at least one camera image, to make it suitable for identifying the target object, and generates a corrected bounding box indicating the corrected object region; and a learning unit that trains a learning model for image recognition using the position information of the corrected bounding box.

[0009] The learning device according to the embodiments of this disclosure reduces the effort required to annotate bounding boxes that indicate appropriate object regions for object retrieval.

[0010] This figure shows an example of the overall configuration of the learning device according to Embodiment 1. This figure shows an example of the hardware configuration of the learning device or image processing device. This is a flowchart of the learning method according to Embodiment 1. This is a flowchart of the subroutines performed by the region correction unit of the learning method according to Embodiment 1. This figure shows a specific example of the configuration of the learning device according to Embodiment 2. This is a schematic diagram showing an example of object detection results (person detection results) by the object detection unit according to Embodiment 2. This is a schematic diagram showing an example of object detection results (segmentation results) by the region correction unit according to Embodiment 2. This is a schematic diagram showing an example of correction results by the region correction unit according to Embodiment 2. This is a schematic diagram showing an example of object detection results by the object detection unit according to Embodiment 2. This is a schematic diagram showing an example of object detection results (skeleton detection results) by the region correction unit according to Embodiment 2. This is a schematic diagram showing an example of correction results by the region correction unit according to Embodiment 2. This is a flowchart of the subroutines performed by the region correction unit of the learning method according to Embodiment 2. This figure shows a specific example of the configuration of the learning device according to Embodiment 3. This is a schematic diagram showing an example of object detection results (person detection results) by the object detection unit according to Embodiment 3. This is a schematic diagram showing an example of object detection results (segmentation results) by the region correction unit according to Embodiment 3. This is a schematic diagram showing an example of correction results by the region correction unit according to Embodiment 3. This is a flowchart showing the subroutines performed by the region correction unit of the learning method according to Embodiment 3. This is a diagram showing an example of the overall configuration of the learning device according to Embodiment 4. This is a schematic diagram showing an example of object detection results (person detection results) by the image correction unit according to Embodiment 4. This is a schematic diagram showing an example of object detection results (segmentation results) by the image correction unit according to Embodiment 4. This is a schematic diagram showing an example of correction results by the image correction unit according to Embodiment 4. This is a flowchart of the learning method according to Embodiment 4. This is a flowchart of the subroutines of the learning method according to Embodiment 4. This is a schematic diagram showing an example of object detection results (person detection results) by the image correction unit according to a modified example (modification 1) of Embodiment 4. This is a schematic diagram showing an example of object detection results (moving object detection results) by the image correction unit according to a modified example (modification 1) of Embodiment 4.This is a schematic diagram showing an example of object detection results (segmentation results) by the image correction unit according to a modified example (modification 1) of Embodiment 4. This is a schematic diagram showing an example of correction results by the image correction unit according to a modified example (modification 1) of Embodiment 4. This is a flowchart of the learning method subroutine according to a modified example (modification 1) of Embodiment 4. This is a schematic diagram showing an example of object detection results (person detection results) by the object detection unit according to a modified example (modification 2) of Embodiments 1 to 4. This is a schematic diagram showing an example of object detection results (segmentation results) by the region correction unit according to a modified example (modification 2) of Embodiments 1 to 4. This is a schematic diagram showing an example of correction results by the region correction unit according to a modified example (modification 2) of Embodiments 1 to 4. This is a flowchart of the learning method subroutine3) of Embodiments 1 to 4. This is a flowchart of the learning method subroutine according to a modified example (modification 4) of Embodiments 1 to 4. This is a diagram showing an example of the configuration of the image processing device according to Embodiment 5. This is a flowchart of the image processing method according to Embodiment 5. This is a diagram showing an example of the configuration of the image processing device according to Embodiment 6.

[0011] Various embodiments of this disclosure will be described in detail below with reference to the drawings. While the following embodiments describe object retrieval technology as an example, the technologies of this disclosure are not limited to this example and are broadly applicable to image recognition technologies that learn models using object location information, such as object tracking technology. In the drawings, identical or similar parts are denoted by the same or similar reference numerals, and redundant descriptions of such parts are omitted. Furthermore, in this disclosure, the term "or" is used to mean inclusive OR unless otherwise specified.

[0012] Embodiment 1. <Configuration> The learning device 10 according to Embodiment 1 of the present disclosure will be described with reference to Figure 1. The learning device 10 is a learning device for object retrieval. As shown in Figure 1, the learning device 10 comprises an object detection unit 11, a region correction unit 12, a learning unit 13, a storage device 14, a storage device 15, and a storage device 16.

[0013] (Object Detection Unit) The object detection unit 11 is a functional unit that detects the region of a desired object from a camera image. For example, the object detection unit 11 detects an object region in which an object such as a person exists from the video stored in the storage device 14 using known techniques. The video includes a plurality of time-series camera images taken by the camera. Camera image is used as a term interchangeable with frame. The object detection unit 11 detects an object from the video using known methods such as R-CNN (Region-Based CNN), YOLO (You Only Look Once), or SSD (Single Shot Multibox Detector), and obtains a bounding box (circumscribing rectangle; first bounding box) that indicates the object region in which the detected object exists. The object detection unit 11 supplies video data, coordinate data indicating the position of the bounding box, and a confidence level, which is the probability that the detected object belongs to a predetermined class, to the area correction unit 12.

[0014] The following explanation will be based on the case where the object is a person. For example, the object domain will be assumed to be the person domain. Note that the object may also be something other than a person, such as a vehicle or an animal.

[0015] (Region Correction Unit) The Region Correction Unit 12 is a functional unit that corrects the bounding box indicating the object region where the target object detected from at least one camera image is located, so that it is suitable for identifying the target object, and generates a corrected bounding box indicating the corrected object region. The person region detected by the object detection unit 11 using known technology may not be suitable for identifying the person. For example, useful information for person search, such as shoes, lower body, hands, arms, and belongings, may be missing. Therefore, the Region Correction Unit 12 corrects the person region detected by the object detection unit 11 into a corrected person region suitable for identifying the person, so that the person region detected by the object detection unit 11 becomes a region suitable for identifying the person. The corrected person region is a region suitable for identifying the person within each camera image. A region suitable for identifying the person means a region in which the bounding box includes as much of the person as possible, while excluding as little as possible any unnecessary objects other than the person. If the person region detected by the object detection unit 11 is suitable for identifying the person, the Region Correction Unit 12 performs a 0 correction.

[0016] Corrected person regions that are not suitable for identifying a person even after correction may be excluded from the dataset. For example, the region correction unit 12 may exclude a corrected person region from the dataset if part of the target person is obscured within that region. However, corrected person regions where part of the target person is obscured may be included in the dataset, assuming use in environments where parts of the body are frequently hidden, such as surveillance in a train station. An example of a case where part of the target person is obscured includes cases where another person is visible in front of the target person. In the context of use in surveillance in a train station, the trained model is used to search for the target person from a group of people captured by surveillance cameras placed in the station. The target person refers to a person detected by the object detection unit, i.e., a person indicated by a bounding box.

[0017] The region correction unit 12 stores coordinate data indicating the position of the bounding box representing the corrected human region in each camera image in the storage device 15.

[0018] (Learning Unit) The learning unit 13 trains a learning model for person search using video data including multiple camera images and corrected person regions. The corrected person region is indicated by the position information (coordinate information) of the corrected bounding box. The learning model is a machine learning model such as a CNN. The learning unit 13 learns the correlation between the corrected person region and the ID of that person. The annotation of IDs for people represented in the camera images may be performed by the learning unit 13 using the publicly known technique described in Non-Patent Literature 1. Alternatively, the learning model may be trained using video data that already has person IDs assigned to it. When ID annotation is performed using the publicly known technique described in Non-Patent Literature 1, unsupervised learning of the learning model becomes possible. The learning unit 13 learns the parameters of the learning model so that the feature quantities obtained from images taken with different cameras, at different times, or in different environments of the same person become similar in the feature space, and the feature quantities of different people become far apart in the feature space. To perform such learning, instance-level consistent learning and cluster-level comparative learning may be performed, as taught in Non-Patent Document 1. The learning unit 13 stores the trained model obtained through training in the memory device 16.

[0019] (Storage Devices) Storage device 14 is a device that stores video data acquired from the camera. Storage device 15 is a device that stores corrected human area data. Storage device 16 is a device that stores trained models.

[0020] <Hardware Configuration> Next, an example of the hardware configuration of the learning device 10 will be described with reference to Figure 2. The learning device 10 can be realized by a processor 101, which is a computer that executes a program stored in memory 102 as shown in Figure 2. The functions of the learning device 10 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in memory 102. The processor 101 realizes the functions of the learning device 10 by reading and executing the program stored in memory 102. Here, examples of memory 102 include non-volatile or volatile semiconductor memories such as RAM (random access memory), ROM (read-only memory), flash memory, EPROM (erasable programmable read-only memory), and EEPROM (electrically erasable programmable read-only memory), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, and DVDs.

[0021] The storage device 103 is, for example, a hard disk, and corresponds to storage devices 14 to 16. The input / output interface 104 is an interface for exchanging information with the outside of the learning device 10.

[0022] Furthermore, the image processing device described later can also be realized with the hardware configuration shown in Figure 2.

[0023] Next, the learning method performed by the learning device 10 will be described with reference to Figures 3 and 4. Figure 3 is a flowchart showing the overall flow of the learning method performed by the learning device 10, and Figure 4 is a flowchart showing step ST2 in Figure 3 in more detail.

[0024] (Step ST1) In step ST1, the object detection unit 11 detects the area of ​​a person from the video data using known techniques.

[0025] (Step ST2) In step ST2, the region correction unit 12 corrects the person region detected by the object detection unit 11 to a region suitable for identifying a person.

[0026] (Step ST211) The subroutine of step ST2 will be explained with reference to Figure 4. In step ST211 in Figure 4, the region correction unit 12 selects one region before correction from among the regions of the person detected by the object detection unit 11.

[0027] (Step ST212) In step ST212, the region correction unit 12 corrects the region selected in step ST211 to a region suitable for identifying a person.

[0028] (Step ST213) In step ST213, the area correction unit 12 generates corrected person area data from the person area corrected in step ST212.

[0029] (Step ST214) In step ST214, the region correction unit 12 completes the process shown in Figure 4 if it has completed the correction for the regions of all people detected by the object detection unit 11. If the correction is not completed, the process returns to step ST211.

[0030] (Step ST3) Returning to Figure 3, in step ST3, the learning unit 13 uses the video data, corrected person region data, and assigned ID as training data to train the learning model.

[0031] Conventional weakly supervised person search requires the assignment of person regions to camera images during training, which is labor-intensive and has not been resolved. The learning device 10 of this disclosure corrects the person regions obtained as a result of person detection and assigns corrected person regions, thereby reducing the labor required for assigning person regions. Furthermore, the learning device 10 of this disclosure does not simply assign the person detection results as person regions, but corrects the person detection results to make them suitable for identifying people, and uses the corrected person regions as training data for training, thereby improving the accuracy of the learning process.

[0032] Embodiment 2. Next, the learning device 10 according to Embodiment 2 will be described with reference to Figures 5 to 8. The learning device 10 according to Embodiment 2 is no different in overall configuration from the learning device 10 according to Embodiment 1, which was described with reference to Figure 1. The learning device 10 according to Embodiment 2 is a more concrete form of the learning device 10 according to Embodiment 1, in that the area correction unit 12 is equipped with an auxiliary area acquisition unit 12-1.

[0033] As shown in Figure 5, the region correction unit 12 of the learning device 10 according to Embodiment 2 includes an auxiliary region acquisition unit 12-1. The auxiliary region acquisition unit 12-1 acquires the region of an object, such as a person, in a way different from the object detection method used by the object detection unit 11. For example, the auxiliary region acquisition unit 12-1 acquires the region and confidence level of an object by instance segmentation using Mask R-CNN (Region-Based CNN) or by skeleton detection using OpenPose or ViTPose. The auxiliary region acquisition unit 12-1 may also acquire the region of a person based on the region of a person detected by the object detection unit 11. In this way, it becomes possible to associate the region of a person selected in step ST222 (output of the object detection unit 11) with the region of a person acquired in step ST221 (output of the auxiliary region acquisition unit 12-1). For example, the auxiliary region acquisition unit 12-1 may acquire the region of a person from the region of a person detected by the object detection unit 11, or the auxiliary region acquisition unit 12-1 may acquire the region of a person from a region that is slightly wider than the region of a person detected by the object detection unit 11.

[0034] The region correction unit 12 performs corrections based on the person region detected by the object detection unit 11 and the person region acquired by the auxiliary region acquisition unit 12-1. As an example, as shown in Figures 6A to 6C, the region correction unit 12 may adopt the result with higher confidence between person detection by the object detection unit 11 (Figure 6A) and instance segmentation by the auxiliary region acquisition unit 12-1 (Figure 6B), and perform corrections (Figure 6C). That is, the bounding box obtained by person detection by the object detection unit 11 may be corrected to include the person region with higher confidence. Figure 6C shows an example in which the bounding box is corrected to surround the region acquired by instance segmentation in Figure 6B, assuming that the confidence level in Figure 6B is higher than that in Figure 6A.

[0035] As another example, as shown in Figures 7A to 7C, the region correction unit 12 may adopt the result with higher confidence from the person detection by the object detection unit 11 (Figure 7A) and the skeleton detection by the auxiliary region acquisition unit 12-1 (Figure 7B), and perform correction (Figure 7C). Since the confidence level from skeleton detection is calculated for each of the multiple key points, the average of the confidence levels of the multiple key points may be calculated as a single confidence level from skeleton detection, and this calculated confidence level may be compared with the confidence level from the person detection by the object detection unit 11. The region correction unit 12 may correct the bounding box from the person detection by the object detection unit 11 so that it includes the person region with the higher confidence level. For example, the region correction unit 12 may correct the bounding box indicating the person region detected by the object detection unit 11 by shrinking or enlarging it so that it includes the result from skeleton detection.

[0036] The region correction unit 12 may use the results of skeleton detection as the correction result. That is, without comparing the reliability, it may correct the bounding box indicating the human region detected by the object detection unit 11 by shrinking or enlarging it to include the results of skeleton detection.

[0037] By correcting the human region using these methods, it becomes possible to correct the human region to include the entire target person. As a prerequisite for such correction, camera images containing occlusion in the target person may be excluded from processing. The exclusion of camera images containing occlusion may be performed by the region correction unit 12. In the case of semantic segmentation, the presence or absence of occlusion can be determined by whether or not multiple pixels belonging to the segment determined to be the human region are located at the edge of the camera image. If multiple pixels are located at the edge of the camera image, it may be determined that occlusion has occurred. In the case of skeleton detection, the presence or absence of occlusion can be determined by whether or not the number of keypoints is above a predetermined threshold. If the number of keypoints is below the predetermined threshold, it may be determined that occlusion has occurred.

[0038] Thus, the region correction unit 12 may include an auxiliary region acquisition unit 12-1 that detects an object region (second object region) from the camera image using a second method different from the detection method (first method) used by the object detection unit 11. In this case, the region correction unit 12 may generate a corrected bounding box based on the result of comparing information (e.g., confidence level) indicating the object region (first object region) detected by the object detection unit 11 with information (e.g., confidence level) indicating the second object region.

[0039] The second method may be a method for detecting a second object region with a second level of confidence, and the region correction unit 12 may generate a corrected bounding box using either the first object region or the second object region detected by the method that shows a higher value among the first level of confidence obtained by the method used by the object detection unit 11 or the second level of confidence obtained by the method used by the region correction unit 12.

[0040] By integrating the results of estimating a person region using a plurality of methods in this manner, it is possible to improve the accuracy of the person region suitable for person search. By adopting the result with higher reliability, it is possible to improve the accuracy of the person region suitable for person search.

[0041] Next, with reference to FIG. 8, a flow executed by the region correction unit 12 of the learning device 10 according to the second embodiment will be described. The flowchart in FIG. 8 shows a flow executed as a subroutine of step ST2 in FIG. 3.

[0042] (Step ST221) In step ST221, the auxiliary region acquisition unit 12-1 of the region correction unit 12 acquires a person region in the camera image by a method different from that of the object detection unit 11.

[0043] (Step ST222) In step ST222, the region correction unit 12 selects one uncorrected region from the person regions detected by the object detection unit 11.

[0044] (Step ST223) In step ST223, the region correction unit 12 corrects the person region based on the person region selected in step ST222 and the person region acquired in step ST221, so that the target person is fully captured without omission and regions other than the target person are not excessively captured. The region correction unit 12, for example, selects a region (output of the auxiliary region acquisition unit 12-1) that has the maximum Intersection over Union (IoU) with the person region selected in step ST222 and is equal to or greater than a threshold value, and then corrects the person region based on the person region (output of the object detection unit 11) selected in step ST222 and the person region (output of the auxiliary region acquisition unit 12-1) acquired in step ST221.

[0045] (Step ST224) In step ST224, the region correction unit 12 generates corrected person region data from the person region corrected in step ST223.

[0046] (Step ST225) In step ST225, if the correction for all person regions detected by the object detection unit 11 has been completed, the region correction unit 12 terminates the process. If the correction has not been completed, the process returns to step ST222.

[0047] Third Embodiment. Next, the learning device 10 according to the third embodiment will be described with reference to FIGS. 9 to 11. The overall configuration of the learning device 10 according to the third embodiment is the same as that of the learning device 10 according to the first embodiment described with reference to FIG. 1. As shown in FIG. 9, the learning device 10 according to the third embodiment is a more specific embodiment of the learning device 10 according to the first embodiment, in that the region correction unit 12 includes an auxiliary region acquisition unit 12-1 and an occupancy rate calculation unit 12-2. The auxiliary region acquisition unit 12-1 has already been described in accordance with the second embodiment.

[0048] As shown in FIG. 10A, the object detection unit 11 detects a bounding box indicating a person region. In the bounding box of FIG. 10A, in addition to the target person P1, a person P2 present behind the target person P1 is captured in the camera image. The region correction unit 12 corrects the bounding box so that the region of the target person P1 is enlarged and the region of the person P2 is reduced. To achieve such an object, the region correction unit 12 includes the auxiliary region acquisition unit 12-1 and the occupancy rate calculation unit 12-2. Note that, of the person P1 and the person P2, the person who is captured more widely within the bounding box detected by the object detection unit 11 is the target person.

[0049] As shown in FIG. 10B, the auxiliary region acquisition unit 12-1 acquires the region of each person from the camera image by instance segmentation.

[0050] The occupancy rate calculation unit 12-2 calculates the occupancy rate of each person acquired by the auxiliary region acquisition unit 12-1 within the bounding box detected by the object detection unit 11. The occupancy rate is the ratio of the number of pixels in each person's region to the total number of pixels in the bounding box.

[0051] The region correction unit 12 corrects the size of the bounding box detected by the object detection unit 11 so that the occupancy rate of the target person P1 is high and the occupancy rate of the other person P2 is low. Figure 10C is a schematic diagram showing the result of this correction. For example, the region correction unit 12 may enlarge the size of the bounding box until the increase rate of pixels of the target person P1 becomes 0. As another example, the region correction unit 12 may enlarge the size of the bounding box if the increase rate of pixels of the target person P1 is greater than the increase rate of pixels of person P2. As yet another example, the region correction unit 12 may reduce the size of the bounding box if the decrease rate of pixels of person P2 is greater than the decrease rate of pixels of the target person P1. The region correction unit 12 corrects the region of the person so that the pixel occupancy rate of the target person P1 is high and the pixel occupancy rate of person P2 is low, by appropriately combining these methods or other methods.

[0052] In this way, the region correction unit 12 detects a first object (e.g., P1) which is the target object and a second object (e.g., P2) which is not the target object from the camera image, and changes the size of the bounding box so that the area of ​​the first object included in the bounding box becomes wider and the area of ​​the second object becomes narrower.

[0053] Next, with reference to Figure 11, the flow performed by the area correction unit 12 of the learning device 10 according to Embodiment 3 will be described. The flowchart in Figure 11 shows the flow performed as a subroutine of step ST2 in Figure 3.

[0054] (Step ST231) In step ST231, the auxiliary region acquisition unit 12-1 of the region correction unit 12 acquires the region of a person in the camera image in a different manner from the object detection unit 11.

[0055] (Step ST232) In step ST232, the region correction unit 12 selects one region from the region of the person detected by the object detection unit 11 that is before correction.

[0056] (Step ST233) In step ST233, the occupancy rate calculation unit 12-2 of the area correction unit 12 calculates the occupancy rate of each person acquired by the auxiliary area acquisition unit 12-1 within the bounding box indicating the person area detected by the object detection unit 11. The area correction unit 12 also corrects the size of the bounding box detected by the object detection unit 11 so that the occupancy rate of the target person P1 is high and the occupancy rate of other people P2 is low.

[0057] (Step ST234) In step ST234, the region correction unit 12 generates corrected person region data from the person region corrected in step ST233.

[0058] (Step ST235) In step ST235, the region correction unit 12 completes the process if it has completed the correction for the regions of all people detected by the object detection unit 11. If the correction is not completed, the process returns to step ST232.

[0059] Embodiment 4. <Configuration> Next, the learning device 10-1 according to Embodiment 4 will be described with reference to Figures 12 to 15. The learning device 10-1 according to Embodiment 4 includes an additional component, an image correction unit 17, compared to the learning device 10 according to Embodiment 1 or 2. The learning device 10-1 may also include a storage device 18 that holds the image data corrected by the image correction unit 17.

[0060] (Image Correction Unit) As shown in Figures 13A to 13C, the image correction unit 17 acquires the area of ​​a person detected by the object detection unit 11 and interpolates the area of ​​person P2 that is reflected behind the target person P1 with the background of other frames in which no person is visible. The front-to-back relationship between people is acquired, for example, by depth estimation using a Depth Prediction Transformer.

[0061] By filling in the areas of other people that appear behind the target person with the background, it is possible to exclude the areas of other people that could negatively impact the learning process for person retrieval.

[0062] <Operation> Next, the operation of the learning device 10-1 according to Embodiment 4 will be described with reference to Figures 14 and 15. Figure 14 is a flowchart showing the overall flow of the learning method performed by the learning device 10-1, and Figure 15 is a flowchart showing step ST1-2 in Figure 14 in more detail.

[0063] (Step ST1) In step ST1, the object detection unit 11 detects the area of ​​a person from the video data using known techniques.

[0064] (Step ST1-2) In step ST1-2, the video correction unit 17 corrects the area of ​​unnecessary objects in the video.

[0065] (Step ST1201) In step ST1201 in Figure 15, the image correction unit 17 acquires the region of a person in a different way than the object detection unit 11. For example, it acquires the region of a person by instance segmentation.

[0066] (Step ST1202) In step ST1202, the video correction unit 17 obtains the front-to-back relationship between adjacent people to the person's region obtained in step ST1201.

[0067] (Step ST1203) In step ST1203, the video correction unit 17 acquires frames from the video captured by the same camera that do not contain any people.

[0068] (Step ST1204) In step ST1204, the video correction unit 17 fills in the area that was determined to be a person other than the foreground in step ST1202 with the image of the frame acquired in step ST1203.

[0069] (Step ST2) In step ST2, the region correction unit 12 corrects the region of the person detected by the object detection unit 11 and whose background has been supplemented by the image correction unit 17 to a region suitable for identifying the person.

[0070] (Step ST3) In Step ST3, the learning unit 13 trains the learning model.

[0071] <Modification of Embodiment 4; Modification 1> In Embodiment 4 described above, the explanation focused on the case where a person is visible in the background of the subject. However, if a moving object other than a person, such as a car or bicycle, is visible in the background of the subject, the moving object may be interpolated in the same way as in Embodiment 4. As shown in Figures 16A to 16D, the area of ​​a moving object other than the subject (e.g., a moving car or person) that is visible behind the subject is interpolated with the background of another frame. The moving object is detected, for example, by background subtraction. The front-to-back relationship between the person and the moving object is obtained, for example, by depth estimation. <Operation>

[0072] (Step ST1205) In this modified example, as a subroutine of step ST1-2 in Figure 14, the video correction unit 17 performs the series of processes shown in Figure 17. In step ST1205, the video correction unit 17 obtains the region of the moving object in the frame, for example, by background subtraction.

[0073] (Step ST1206) In step ST1206, the video correction unit 17 determines the attributes of the moving object (person, car, bicycle, etc.) acquired in step ST1205. For example, the attributes are determined by instance segmentation.

[0074] (Step ST1207) In step ST1207, the image correction unit 17 acquires the front-to-back relationship between adjacent moving objects for the region of the moving object acquired in step ST1205.

[0075] (Step ST1208) In step ST1208, the video correction unit 17 acquires frames from the video captured by the same camera that do not contain any moving objects.

[0076] (Step ST1209) In step ST1209, the video correction unit 17 interpolates the area of ​​the moving object that was determined to be behind the person that was determined to be at the very front in steps ST1206 and ST1207 using the frame acquired in step ST1208.

[0077] In this way, the image correction unit 17 detects the first object and the second object from the first camera image using a second method different from the object detection method (first method) by the object detection unit 11, based on a first camera image that includes the first object, which is the target object, and a second object that is not the target object and is located behind the first object, and a second camera image that does not include either the first object or the second object, and complements the area of ​​the second object included in the bounding box with the background of the second camera image.

[0078] <Modifications of Embodiments 1-4; Modification 2> As shown in Figures 18A-18C, the region correction unit 12 may correct the bounding box surrounding the region of the target person so that it also includes the target person's possessions. Whether an object touching the target person is a possession is determined, for example, by a scene graph. By correcting the region to include the area of ​​the target person's possessions in this way, the region may better capture the characteristics of the target person. When searching for a person to be searched, including their possessions, during inference using a learning model, it is beneficial to train the learning model with the target person including their possessions to build a better learning model. However, depending on the environment to which the technology of this disclosure is applied, the region may be corrected to exclude possessions. For example, a supermarket shopper may have a shopping cart indoors but not outdoors. In this case, correcting the results of person detection using video footage captured by a camera installed indoors to exclude the cart results in a region that captures the characteristics of the target person and is robust to changes in possessions.

[0079] <Operation> Next, the operation of the region correction unit 12 will be explained with reference to Figure 19. Figure 19 is a flowchart showing the process executed as a subflow of step ST2 shown in Figure 3.

[0080] (Step ST241) In step ST241, the region correction unit 12 acquires the region of an object within the frame, for example, by instance segmentation.

[0081] (Step ST242) In step ST242, the region correction unit 12 selects one region from the region of the person detected by the object detection unit 11 that is before correction.

[0082] (Step ST243) In step ST243, the region correction unit 12 obtains a pair of the target person and their possessions, for example by generating a scene graph.

[0083] (Step ST244) In step ST244, the area correction unit 12 corrects the area of ​​the person selected in ST242 and the area of ​​the possessions acquired in step ST243 so that the person and possessions are captured without omission and other areas are not captured excessively.

[0084] (Step ST245) In step ST245, the area correction unit 12 generates corrected person area data from the person area corrected in step ST244.

[0085] (Step ST246) In step ST246, the region correction unit 12 completes the process if it has completed the correction for the regions of all people detected by the object detection unit 11. If the correction is not completed, the process returns to step ST242.

[0086] <Modifications of Embodiments 1-4; Modification 3> The region correction unit 12 may correct the person region using a multimodal generation AI. In this case, the region correction unit 12 provides the multimodal generation AI with a frame image, the person region (coordinate information) detected by the object detection unit 11, and a text prompt for correction instructions as input, and obtains the corrected person region (coordinate information) as output from the multimodal generation AI. The correction instruction text may be an instruction such as, "For the person in the given region, output rectangular coordinate information that includes the person's belongings and their entire body," or "For the person in the given region, output rectangular coordinate information that includes the person's entire body." By using the multimodal generation AI in this way, it becomes possible to operate with a wide range of options depending on the situation. The multimodal generation AI may be stored in the storage device of the learning device 10, or it may be stored in an external storage device of the learning device 10. If the multimodal generation AI is stored in an external storage device of the learning device 10, the learning device 10 is equipped with a communication interface (not shown).

[0087] <Operation> Next, the operation of the region correction unit 12 will be explained with reference to Figure 20. Figure 20 is a flowchart showing the process executed as a subflow of step ST2 shown in Figure 3.

[0088] (Step ST251) In step ST251, the region correction unit 12 selects one region from the region of the person detected by the object detection unit 11 that is before correction.

[0089] (Step ST252) In step ST252, the region correction unit 12 inputs the frame image, the region selected in step ST251, and the correction instruction text to the multimodal generation AI to obtain the corrected region of the person.

[0090] (Step ST253) In step ST253, the area correction unit 12 generates corrected person area data from the person area corrected in step ST252.

[0091] (Step ST254) In step ST254, the area correction unit 12 completes the process if it has completed the correction for the areas of all people detected by the object detection unit 11. If the correction is not completed, the process returns to step ST251.

[0092] <Modifications of Embodiments 1 to 4; Modification 4> The region correction unit 12 may use multimodal generation AI to determine whether the person region detected by the object detection unit 11 is a suitable region for identifying a person. If the multimodal generation AI outputs a response that it is not suitable, the region correction unit 12 removes the camera image that was determined to be unsuitable from the dataset. By utilizing multimodal generation AI in this way, it becomes possible to make judgments that would be difficult with rule-based methods.

[0093] In this case, the region correction unit 12 provides the multimodal generation AI with a camera image including the person region detected by the object detection unit 11 and a text prompt for judgment instructions as input, and receives a response from the multimodal generation AI indicating whether it is suitable or unsuitable.

[0094] <Operation> Next, the operation of the region correction unit 12 will be explained with reference to Figure 21. Figure 21 is a flowchart showing the process executed as a subflow of step ST2 shown in Figure 3.

[0095] (Step ST261) In step ST261, the region correction unit 12 selects one region from the region of the person detected by the object detection unit 11 that is before correction.

[0096] (Step ST262) In step ST262, the region correction unit 12 inputs the region selected in step ST261 and the judgment instruction text to the multimodal generation AI and determines whether the region selected in step ST261 is a region suitable for identifying a person.

[0097] (Step ST263) In step ST263, the region correction unit 12 uses only the regions that were determined to be suitable for identifying a person in step ST262 as corrected person region data.

[0098] (Step ST264) In step ST264, the area correction unit 12 completes the process if it has completed the correction for the areas of all people detected by the object detection unit 11. If the correction is not completed, the process returns to step ST261.

[0099] Embodiment 5. <Configuration> The image processing device 20 according to Embodiment 5 will be described below. In the person search method, which performs person detection and person identification in two steps, person detection is also performed during inference, but the detected area may not be suitable for identifying the person. Therefore, the same correction as during training is performed on the results of person detection performed during inference. Note that this embodiment is a limited form of Embodiment 6, which will be described later.

[0100] As shown in Figure 22, the image processing device 20 uses a trained model generated by the learning device 10 to search for an object identical to the query object represented in the reference image from among the search target objects captured by the imaging device 30. There may be multiple imaging devices 30. To achieve this objective, the image processing device 20 includes an image acquisition unit 21, an object detection unit 22, a region correction unit 23, a region estimation unit 24, and an image reading unit 25, as shown in Figure 20. The image processing device 20 acquires the reference image by reading it from the storage device 40.

[0101] (Image acquisition unit) The image acquisition unit 21 acquires video footage including multiple camera images captured by the imaging device 30. The imaging device 30 is the camera to be searched.

[0102] (Object Detection Unit) The object detection unit 22 performs the same functions as the object detection unit 11 described above. The object detection unit 22 detects a bounding box (second bounding box) indicating the object region where the object to be searched is located from at least one camera image.

[0103] (Region Correction Unit) The region correction unit 23 performs the same functions as the region correction unit 12 described above. The region correction unit 23 corrects the second bounding box to be suitable for identifying the object to be searched and generates a corrected bounding box (second corrected bounding box) that indicates the corrected object region.

[0104] (Image reading unit) The image reading unit 25 obtains a reference image from the corrected person area data for inference stored in the storage device 40. An example of a reference image is an image of the query person.

[0105] (Region Estimation Unit) The region estimation unit 24 uses a trained model generated by the learning device 10 to estimate the region of the same person as the person in the reference image from among the candidate person regions (object regions) acquired by the region correction unit 23 in the image processing device 20.

[0106] More specifically, the region estimation unit 24 acquires a reference image from the image reading unit 25, while simultaneously acquiring multiple search target images (images of regions enclosed by bounding boxes) that have undergone region correction from the region correction unit 23. The region estimation unit 24 inputs these acquired images into a trained model and obtains feature quantities (feature vectors) for each image from the trained model. The region estimation unit 24 calculates the similarity, such as the cosine similarity, between the feature quantities of the reference image and the feature quantities of the multiple search target images. The region estimation unit 24 outputs the information of the multiple search target images in order of similarity. If there are search target images with a similarity of more than a predetermined threshold to the reference image, the region estimation unit 24 may estimate that the person in the search target images with a similarity of more than that threshold is the same person as the person in the reference image and output the estimation result. The output results are displayed on a display device (not shown) via an input / output interface.

[0107] <Operation> Next, the operation of the image processing device 20 will be explained with reference to Figure 23.

[0108] (Step ST4) In step ST4, the image acquisition unit 21 acquires the camera image to be searched from the imaging device 30.

[0109] (Step ST5) In step ST5, the object detection unit 22 detects the area of ​​a person from the video data using known techniques.

[0110] (Step ST6) In step ST6, the region correction unit 23 corrects the person region detected by the object detection unit 22 to a region suitable for identifying a person.

[0111] (Step ST7) In step ST7, the region estimation unit 24 uses the trained model learned by the learning device 10 to estimate the region of the same person as the reference image from among the candidate person regions acquired by the region correction unit 23 in the image processing device 20. The region estimation unit 24 inputs the reference image and multiple search target images that have undergone region correction into the trained model and obtains feature quantities (feature vectors) for each image from the trained model. The region estimation unit 24 calculates the similarity from the feature quantities (feature vectors) of each image from the trained model and estimates the region of the same person as the reference image.

[0112] Embodiment 6. The image processing device 20A according to Embodiment 6 will be described below. Embodiment 5 described an embodiment in which object detection and region correction are performed on an image acquired from the imaging device 30 to perform real-time inference (object search). However, inference does not need to be performed in real time and may be performed after a certain period of time has elapsed since imaging. Embodiment 6 describes an embodiment that enables such delayed inference. Embodiment 6 will be described with reference to Figure 24.

[0113] As shown in Figure 24, the image processing device 20A, like the image processing device 20 of Embodiment 5, includes an image acquisition unit 21, an object detection unit 22, and a region correction unit 23. Unlike the image processing device 20 of Embodiment 5, the image processing device 20A does not include a region estimation unit 24 and an image reading unit 25. Instead of the region estimation unit 24 and the image reading unit 25, the image processing device 20A includes an image storage unit 26.

[0114] The image storage unit 26 is a functional unit that stores the image after region correction has been performed by the region correction unit 23, i.e., the corrected bounding box image showing the corrected object region, as corrected person region data for inference in the storage device 40. The storage device 40 is implemented using RAM or ROM and stores the saved image group temporarily or long-term. The saved image group is used when performing tasks such as person search or person tracking.

[0115] The image processing device 20A performs steps ST4 to ST6 in the same manner as the operation of the image processing device 20 described with reference to Figure 23. After performing step ST6, the image processing device 20A performs a process to save the image after area correction to the storage device 40, instead of step ST7.

[0116] Furthermore, it is possible to combine embodiments, or to modify or omit each embodiment as appropriate.

[0117] The object search technology disclosed herein can be used to identify a person who is the subject of a query from among many people captured by cameras installed in facilities such as buildings, train stations, and roads.

[0118] 10 Learning device, 10-1 Learning device, 11 Object detection unit, 12 Region correction unit, 12-1 Auxiliary region acquisition unit, 12-2 Occupancy rate calculation unit, 13 Learning unit, 14 Storage device, 15 Storage device, 16 Storage device, 17 Image correction unit, 18 Storage device, 20 Image processing device, 21 Image acquisition unit, 22 Object detection unit, 23 Region correction unit, 24 Region estimation unit, 25 Image reading unit, 26 Image storage unit, 30 Imaging device, 40 Storage device, 101 Processor, 102 Memory, 103 Storage device, 104 Input / output interface.

Claims

1. A learning device comprising: a region correction unit that corrects a first bounding box indicating the object region where a target object is located, detected from at least one camera image, to make it suitable for identifying the target object, and generates a corrected bounding box indicating the corrected object region; and a learning unit that trains a learning model for image recognition using the position information of the corrected bounding box.

2. The learning device according to claim 1, further comprising: an object detection unit that detects a bounding box indicating a first object region with a first level of confidence from the camera image using a first method, as the first bounding box.

3. The learning device according to claim 2, wherein the region correction unit includes an auxiliary region acquisition unit that detects a second object region from the camera image by a second method different from the first method, and generates the corrected bounding box based on the result of comparing information indicating the first object region with information indicating the second object region.

4. The learning device according to claim 3, wherein the second method is a skeleton detection method for detecting the skeleton of the target object, and the region correction unit corrects the bounding box so that the corrected bounding box includes the detected skeleton.

5. The learning device according to claim 3, wherein the second method is a method for detecting the second object region with a second confidence level, and the region correction unit generates the corrected bounding box using either the first object region or the second object region detected by the method showing the higher value of the first confidence level or the second confidence level.

6. The learning device according to claim 5, wherein the second method is instance segmentation, and the region correction unit corrects the corrected bounding box so that it surrounds the object region detected by the instance segmentation when the second reliability of the second method is higher than the first reliability of the first method.

7. The learning device according to any one of claims 3 or 5, wherein the region correction unit detects a first object which is a target object and a second object which is not a target object from the camera image using the second method, and changes the size of the bounding box such that the area of ​​the first object included in the bounding box becomes wider and the area of ​​the second object becomes narrower.

8. The learning device according to claim 2, further comprising: a learning device according to claim 2, wherein the at least one camera image includes a plurality of camera images including a first camera image and a second camera image, the first camera image includes a first object which is a target object and a second object which is not a target object and is located behind the first object, the second camera image does not include either the first object or the second object, and an image correction unit which detects the first object and the second object from the first camera image by a second method different from the first method, and complements the area of ​​the second object included in the bounding box with the background of the second camera image.

9. The learning device according to claim 8, wherein the region correction unit further comprises an auxiliary region acquisition unit that detects a second object region from the first camera image after interpolation using the second method, and generates the corrected bounding box based on the result of comparing information indicating the first object region with information indicating the second object region.

10. The learning device according to claim 9, wherein the second method is a method for detecting the second object region with a second confidence level, and the region correction unit generates the corrected bounding box using either the first object region or the second object region detected by the method showing the higher value of the first confidence level or the second confidence level.

11. The learning device according to any one of claims 8 to 10, wherein the first object is a person and the second object is a moving object other than a person.

12. The learning device according to any one of claims 1 to 11, wherein the target object is a person, and the area correction unit generates the corrected bounding box such that the person's possessions are included in the corrected bounding box.

13. The learning device according to claim 1, wherein the region correction unit provides the camera image, the bounding box information, and a text prompt instructing the multimodal generation AI to correct the bounding box so that it is within a range suitable for identifying an object, and obtains the response from the multimodal generation AI as the corrected bounding box.

14. The learning device according to claim 1, wherein the area correction unit provides the multimodal generation AI with the camera image, the information of the first bounding box, and a text prompt instructing the AI ​​to determine whether the range of the first bounding box is suitable for identifying an object, and the learning unit performs the training using the image that the multimodal generation AI has determined to be suitable.

15. An image processing apparatus comprising: a region correction unit that corrects a second bounding box indicating an object region in which a second target object detected from at least one camera image is located to be suitable for identifying the second target object, thereby generating a second corrected bounding box indicating the corrected object region; and a region estimation unit that inputs an image of the corrected object region indicated by the second corrected bounding box and a reference image to a learning model trained by a learning device according to any one of claims 1 to 14, obtains feature quantities from each image from the trained learning model, calculates the similarity between the obtained feature quantities, and estimates the region of the same person as the person in the reference image.

16. An image processing apparatus comprising: a region correction unit that corrects a bounding box indicating an object region in which a target object detected from at least one camera image is located to be suitable for identifying the target object, and generates a corrected bounding box indicating the corrected object region; and an image storage unit that stores an image of the corrected bounding box in a storage device.

17. A learning method performed by a learning device comprising a region correction unit and a learning unit, comprising the steps of: the region correction unit correcting a first bounding box indicating the object region where a target object is located, detected from at least one camera image, to make it suitable for identifying the target object, thereby generating a corrected bounding box indicating the corrected object region; and the learning unit training a learning model for image recognition using the positional information of the corrected bounding box.

18. A learning program that causes a computer to perform the following functions: a function to correct a first bounding box indicating the object region where a target object is located, detected from at least one camera image, to be suitable for identifying the target object, thereby generating a corrected bounding box indicating the corrected object region; and a function to train a learning model for image recognition using the positional information of the corrected bounding box.

Images