Detection device, detection method, and recording medium

Abstract

This detection device is provided with a target region setting unit and a first detection unit. The target region setting unit processes a first target frame image, thereby setting a target region in a second target frame image. The first target frame image is a frame image included in a plurality of frame images that are a time-series. The second target frame image is a frame image captured after the first target frame image among the plurality of frame images. The target region is a region that should be subjected to processing for detecting a target object that is not included in the first target frame image. The first detection unit executes detection processing for the target object on the target region in the second target frame image.

Description

Detection device, detection method, and recording medium

[0001] The present invention relates to a detection device, a detection method, and a program.

[0002] In monitoring by a camera or the like, it is desirable to detect a target object such as a person with low latency.

[0003] Patent Document 1 describes predicting the area of a bounding box in the latest frame and specifying the position of the bounding box of the object shown, targeting the area of the predicted bounding box.

[0004] Japanese Patent Application Laid-Open No. 2024-135463

[0005] The technique described in Patent Document 1 above predicts the area of the bounding box in the latest frame based on the history information of the bounding box for a common object. Therefore, there is a problem that it is difficult to detect an object that newly appears in the latest frame with low latency.

[0006] An example of the object of the present invention is to provide a detection device, a detection method, and a program capable of detecting a target object that newly appears in a video with low latency in view of the above-described problems.

[0007] The detection device according to one aspect of the present disclosure includes a target area setting means for setting a target area for performing processing for detecting a target object that is not included in the first target frame image in a second target frame image captured after the first target frame image among the plurality of frame images by processing the first target frame image included in the plurality of time-series frame images, and a first detection means for executing the detection processing of the target object on the target area in the second target frame image.

[0008] In one aspect of the detection method of this disclosure, one or more computers process a first target frame image included in a plurality of time-series frame images to set a target region in a second target frame image captured after the first target frame image among the plurality of frame images, in which processing should be performed to detect a target object not included in the first target frame image, and execute the target object detection processing on the target region in the second target frame image.

[0009] A recording medium in one aspect of this disclosure is a computer-readable recording medium on which a program is recorded, the program causing the computer to function as a target area setting means for setting a target area in a second target frame image captured after the first target frame image among the multiple frame images, by processing a first target frame image included in a plurality of frame images in a time series, for which processing should be performed to detect a target object not included in the first target frame image, and as a first detection means for performing the target object detection processing on the target area in the second target frame image.

[0010] According to one aspect of the present invention, a detection device, a detection method, and a program capable of detecting newly appearing target objects in a video with low latency can be provided.

[0011] This diagram shows an overview of the detection device related to this disclosure. This diagram shows an overview of the detection method related to this disclosure. This diagram shows an overview of the system related to this disclosure. This diagram illustrates the functional configuration of the detection device related to this disclosure. This diagram illustrates a computer for realizing the detection device. This is a flowchart illustrating the processing flow executed by the detection device. This diagram illustrates a detected attention object. This diagram illustrates a specified attention area. This diagram illustrates a set target area. This diagram illustrates a trained model used by the target area setting unit. This diagram illustrates the edge of the second target frame image. This diagram illustrates the detection of a target object. This diagram illustrates a trained model used by the first detection unit. This diagram illustrates the relationship between the attention area and the target area when the position of the attention object changes over time. This diagram illustrates the operating environment of the detection device. This diagram illustrates the configuration of a detection device equipped with a history information generation unit. This diagram illustrates a trained model used by the target area setting unit. This diagram illustrates a trained model used by the second detection unit. This diagram explains a method for setting the target area using three-dimensional information. This diagram illustrates the operating environment of the detection device.

[0012] In this disclosure, the drawings are associated with one or more embodiments. In all drawings, similar components are denoted by the same reference numerals, and their descriptions are omitted where appropriate.

[0013] (First Embodiment) Figure 1 is a diagram showing an overview of the detection device 10 according to the present disclosure. The detection device 10 comprises a target area setting unit 130 and a first detection unit 150. The target area setting unit 130 sets a target area in the second target frame image by processing a first target frame image. The first target frame image is a frame image included in a plurality of frame images in a time series. The second target frame image is a frame image captured after the first target frame image among the plurality of frame images. The target area is an area where processing should be performed to detect a target object not included in the first target frame image. The first detection unit 150 performs target object detection processing on the target area in the second target frame image.

[0014] This detection device 10 can detect newly appearing target objects in the video with low latency.

[0015] Figure 2 is a diagram illustrating an overview of the detection method according to the present disclosure. The detection method according to the present disclosure includes steps S10 and S20. In step S10, one or more computers process a first target frame image to set a target region in a second target frame image. The first target frame image is a frame image included in a plurality of time-series frame images. The second target frame image is a frame image captured after the first target frame image among the plurality of frame images. The target region is a region in which processing should be performed to detect a target object not included in the first target frame image. In step S20, one or more computers perform a target object detection process on the target region in the second target frame image.

[0016] This detection method allows for the detection of newly appearing target objects within a video with low latency.

[0017] Figure 3 is a diagram illustrating the overview of the system 20 according to this disclosure. The system 20 comprises a detection device 10 and an imaging device 22 that generates a plurality of time-series frame images. The detection device 10 comprises a target area setting unit 130 and a first detection unit 150. The target area setting unit 130 sets a target area in the second target frame image by processing a first target frame image. The first target frame image is a frame image included in a plurality of time-series frame images. The second target frame image is a frame image captured after the first target frame image among the plurality of frame images. The target area is an area where processing should be performed to detect a target object not included in the first target frame image. The first detection unit 150 performs target object detection processing on the target area in the second target frame image.

[0018] According to this system 20, newly appearing target objects in the video can be detected with low latency.

[0019] The following describes detailed examples of the detection device 10, detection method, and system 20 related to this disclosure.

[0020] Figure 4 is a diagram illustrating the functional configuration of the detection device 10 according to the present disclosure. In the example of Figure 4, the detection device 10 further comprises an input receiving unit 120 and an output control unit 170.

[0021] The detection device 10 has the function of detecting a target object from a series of frame images. The series of frame images are captured at predetermined intervals. The series of frame images constitutes a video, for example. The frame rate of this video is not particularly limited, but is, for example, between 1 fps and 240 fps. The target object is, for example, a moving object such as a person, animal, or vehicle.

[0022] The hardware configuration of the detection device 10 will be described below. Each functional component of the detection device 10 (target area setting unit 130, input receiving unit 120, first detection unit 150, and output control unit 170) may be implemented by hardware that realizes each functional component (e.g., hardwired electronic circuits), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). The case in which each functional component of the detection device 10 is implemented by a combination of hardware and software will be described further below.

[0023] Figure 5 illustrates a computer 1000 for implementing the detection device 10. The computer 1000 is any computer. For example, the computer 1000 may be an SoC (System on Chip), a Personal Computer (PC), a server machine, a tablet terminal, or a smartphone. The computer 1000 may be a dedicated computer designed to implement the detection device 10, or it may be a general-purpose computer. Furthermore, the detection device 10 may be implemented with a single computer 1000, or it may be implemented by a combination of multiple computers 1000.

[0024] Computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. However, the method of connecting the processor 1040 and the other components is not limited to bus connection. The processor 1040 is a variety of processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main memory device implemented using RAM (Random Access Memory), etc. The storage device 1080 is an auxiliary storage device implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0025] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as a keyboard and output devices such as a display are connected to the input / output interface 1100. The method by which the input / output interface 1100 connects to the input and output devices may be wireless or wired.

[0026] The network interface 1120 is an interface for connecting the computer 1000 to a network. Examples of such communication networks include LANs (Local Area Networks) and WANs (Wide Area Networks). The network interface 1120 may connect to the network via a wireless connection or a wired connection.

[0027] The storage device 1080 stores program modules that realize each functional component of the detection device 10. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to each program module.

[0028] The target area setting unit 130 and the first detection unit 150 acquire multiple frame images in chronological order. For example, the target area setting unit 130 and the first detection unit 150 acquire frame images captured by the imaging device 22 in real time from the imaging device 22. The imaging device 22 is, for example, an RGB camera or an infrared camera. The multiple frame images may be obtained by capturing images of the outdoors or by capturing images of the indoors. The imaging device 22 may be connected to the input / output interface 1100 or network interface 1120 of the computer 1000 that implements the detection device 10. For example, the multiple frame images are assigned frame numbers in chronological order. That is, newer frame images are assigned larger frame numbers. The target area setting unit 130 and the first detection unit 150 process the acquired frame images in the order they were acquired. Frame images acquired by the target area setting unit 130 are treated as first target frame images, and frame images acquired by the first detection unit 150 are treated as second target frame images.

[0029] Figure 6 is a flowchart illustrating the processing flow performed by the detection device 10. Figure 7 is an example of a detected attention object 53. Figure 8 is an example of a identified attention area 56. Figure 9 is an example of a set target area 54.

[0030] In this embodiment, the target area setting unit 130 detects the attention object 53 contained in the first target frame image 51. The target area setting unit 130 sets the target area 54 using the positional information of the detected attention object 53 within the first target frame image 51. More specifically, the target area setting unit 130 identifies the attention area 56 in the first target frame image 51 using the detection result of the attention object 53. Then, in this embodiment, the target area setting unit 130 sets the area corresponding to the attention area 56 in the second target frame image 52 as the target area 54. The first detection unit 150 performs the detection process for the target object 55 in the target area 54 within the second target frame image 52. This will be explained in detail below.

[0031] A attention object 53 is, for example, an obstruction that may hide the target object 55 in the frame image. Examples of attention objects 53 include buildings (buildings, walls, utility poles, etc.), plants (hedges, trees, etc.), and vehicles. The attention region 56 can be said to be an area in the scene captured in the first target frame image 51 where there is a high probability that the target object 55 will newly appear. For example, a target object 55 appearing from behind a attention object 53 will only appear in the frame image around the attention object 53. By setting the target region 54 using the positional information of the detected attention object 53 within the first target frame image 51, a target object 55 appearing from a blind spot caused by the attention object 53 can be detected with low latency.

[0032] Furthermore, the target area setting unit 130 does not need to detect the target object 55 in order to set the target area 54. The target area setting unit 130 can set the target area 54 regardless of the position of the target object 55 in the first target frame image 51. The target object 55 may or may not be present within the attention area 56 of the identified first target frame image 51.

[0033] In this embodiment, it is preferable that the position of the object of attention 53 within the field of view of the imaging device 22, that is, within multiple frame images, does not change over time. The imaging device 22 may be, for example, a surveillance camera or a fixed camera.

[0034] As described above, the target area setting unit 130 acquires the first target frame image 51 (step S101). In step S102, the target area setting unit 130 detects the attention object 53 in the first target frame image 51. Figure 10 is a diagram illustrating a trained model 111 used by the target area setting unit 130. As an example, the target area setting unit 130 can detect the attention object 53 contained in the first target frame image 51 by inputting the first target frame image 51 into the trained model 111 which detects the attention object 53 in the input frame image. Multiple attention objects 53 may be detected for a single first target frame image 51. By using the trained model 111, the attention object 53 can be detected with high accuracy.

[0035] The trained model 111 accepts a frame image as input. The trained model 111 then outputs attention object information regarding the attention object 53 contained in the input frame image. The attention object information indicates at least the position of the attention object 53 within the input frame image. The information indicating the position of the attention object 53 is, for example, its coordinates within the frame image. The attention object information may further indicate the size of the attention object 53 within the input frame image. The attention object information may, for example, include information indicating the position and size of a box surrounding the attention object 53 in the input frame image. Figure 7 shows a rectangle surrounding the attention object 53 as the box. Specifically, the information indicating the position and size of the box may be the coordinates of a first vertex and a second vertex of the box. The first and second vertices of the box are diagonally opposite each other within the box. Similar examples of information indicating the position and size of the box are given below. The attention object information may further include a label indicating the type of attention object 53 detected (tree, bus, etc.).

[0036] The trained model 111 may be composed of a neural network. The trained model 111 can be obtained, for example, by performing machine learning using training data that associates frame images with ground truth data. As ground truth data, information indicating the position of the attention object 53 in the frame image can be used. The ground truth data may further include information indicating the size of the attention object 53 in the frame image, or it may include the label of the attention object 53. In machine learning, the parameters of the trained model 111 are updated so that the output of the trained model 111 approaches the ground truth data associated with the frame image input to the trained model 111.

[0037] In step S102, the target area setting unit 130 further identifies a attention area 56 using the detection result of the attention object 53 in the first target frame image 51. The target area setting unit 130 can identify a predetermined area as the attention area 56 based on the position of the attention object 53. That is, the target area setting unit 130 can identify a attention area 56 based on the predetermined relationship between the position of the attention object 53 and the position of the attention area 56, and information indicating the position of the detected attention object 53. Note that two or more attention areas 56 spaced apart from each other may be identified for a single attention object 53. As a specific example, the target area setting unit 130 identifies two attention areas 56 that sandwich the position of the attention object 53 shown in the attention object information.

[0038] If the attention object information includes information indicating the size of the attention object 53, the target area setting unit 130 may determine the size of the attention area 56 based on that size. That is, the target area setting unit 130 can identify the attention area 56 based on the predetermined relationship between the size of the attention object 53 and the size of the attention area 56, and the information indicating the size of the detected attention object 53. If the attention object information does not include information indicating the size of the attention object 53, the target area setting unit 130 identifies the attention area 56 with a predetermined size.

[0039] The target area setting unit 130 may further use labels indicating the type of attention object 53 to set the attention area 56. That is, the target area setting unit 130 may further use labels indicating the type of attention object 53 to set the target area 54. In this case, the attention object information further includes labels for the attention object 53. The position of the attention area 56 relative to the position of the attention object 53 is predetermined for each label. Based on the label of each attention object 53, the target area setting unit 130 identifies the relationship between the position of the attention object 53 and the position of the attention area 56 that should be used to identify the attention area 56. In addition, the size of the attention area 56 relative to the size of the attention object 53 may be predetermined for each label. That is, based on the label of each attention object 53, the target area setting unit 130 may identify the relationship between the size of the attention object 53 and the size of the attention area 56 that should be used to identify the attention area 56. The target area setting unit 130 further uses labels indicating the type of object 53 to set a attention area 56, thereby identifying areas where the target object 55 is likely to appear, according to the type of object 53.

[0040] The relationship between the size of the attention object 53 and the size of the attention area 56 may be fixed or variable. In the example in Figure 4, the detection device 10 includes an input receiving unit 120. The input receiving unit 120 receives input regarding the size of the target area 54. This allows the user to adjust the balance between the processing time of the first detection unit 150 and the detection accuracy of the target object 55. The input regarding the size of the target area 54 is, for example, an input to specify the size of the target area 54.

[0041] The input regarding the size of the target area 54 can also be said to be the input regarding the size of the attention area 56. For example, the user of the detection device 10 inputs a value for determining the size of the attention area 56 to the detection device 10 using an input device connected to the computer 1000. The input reception unit 120 acquires the input value. This value may be a value directly indicating the size of the attention area 56 or may be a magnification with respect to the reference size of the attention area 56. The reference size of the attention area 56 is a size determined in advance with respect to the size of the attention object 53. The input reception unit 120 may acquire a value for determining the size of the target area 54 for each label of the attention object 53.

[0042] Based on the input regarding the size of the target area 54 received by the input reception unit 120, the target area setting unit 130 specifies the size of the attention area 56. Then, the attention area 56 having the specified size is specified.

[0043] In the example of FIG. 8, the dashed square is a box surrounding the attention object 53, and the area surrounded by the solid line square is the attention area 56. In FIG. 8, the attention area 56 is specified so as to sandwich at least a part of the attention object 53. The box surrounding the attention object 53 and the attention area 56 may overlap.

[0044] In step S103, based on the position and size of the specified attention area 56, the target area setting unit 130 generates target area information indicating the position and size of the target area 54. The target area information is, for example, information indicating the position and size of a box showing the outer edge of the target area 54. That is, the inside of this box is treated as the target area. In the present embodiment, the target area setting unit 130 sets the area corresponding to the attention area 56 as the target area 54 in the second target frame image 52. That is, the position and size of the attention area 56 with respect to the first target frame image 51 are made the same as the position and size of the target area 54 with respect to the second target frame image 52. In FIG. 9, a plurality of solid line squares are shown as a plurality of boxes indicating the outer edges of the plurality of target areas 54 with respect to the outer edge 59 of the frame image.

[0045] Further, regardless of the detection result of the attention object 53 and the attention area 56, the target area setting unit 130 may include the edge of the second target frame image 52 in the target area 54. FIG. 11 is a diagram illustrating the edge 57 of the second target frame image 52. In FIG. 11, the edge 57 is hatched. Specifically, the area from the edge of the second target frame image 52 to a predetermined distance can be defined as the edge 57. By including the edge 57 of the second target frame image 52 in the target area 54, a target object 55 that frames in from outside the frame image, that is, outside the imaging angle of the imaging device 22, can be detected with low latency.

[0046] In step S103, when the target area setting unit 130 generates the target area information, the first detection unit 150 acquires the target area information from the target area setting unit 130.

[0047] As described above, when the edge 57 of the second target frame image 52 is included in the target area 54, the target area information can include information indicating the width from the edge.

[0048] In step S104, the first detection unit 150 acquires the second target frame image 52 as described above. As can be seen from FIG. 6, this step S104 is executed after step S101. The first detection unit 150 can acquire the latest frame image at that timing. That is, the first detection unit 150 acquires a frame image captured after the first target frame image 51 processed in step S102 as the second target frame image 52. It can be said that the first detection unit 150 processes the frame image of frame number N + k (second target frame image 52) as follows based on the target area 54 specified by processing the frame image of frame number N (first target frame image 51).

[0049] Figure 12 illustrates the detection of the target object 55. In step S105, the first detection unit 150 performs the detection process for the target region 54 within the second target frame image 52. In other words, the first detection unit 150 does not need to perform the detection process for the entire second target frame image 52. By performing the detection process on a portion of the second target frame image 52, the first detection unit 150 can detect the target object 55 with lower latency compared to performing the detection process on the entire second target frame image 52.

[0050] The first detection unit 150 can detect a target object 55 contained in the target region 54 of the second target frame image 52 by inputting the second target frame image 52 and target region information into a trained model that detects a target object 55 in the input frame image. Figure 13 is a diagram illustrating a trained model 151 used by the first detection unit 150. The trained model 151 accepts a frame image and target region information as input. The trained model 151 detects a target object 55 in the region indicated by the input target region information from the input frame image. The trained model 151 outputs target object information relating to the target object 55 that exists within the target region 54 indicated by the target region information from the input frame image. The trained model 151 does not need to detect target objects 55 that exist outside the target region 54.

[0051] The target object information is information that indicates at least the position of the target object 55 within the frame image. The information indicating the position of the target object 55 is, for example, the coordinates within the frame image. The target object information may also be information that indicates the position and size of the target object 55. The information indicating the position and size of the target object 55 may be, for example, information that indicates the position and size of the box 58 that surrounds the target object 55 in the frame image.

[0052] The trained model 151 may be composed of a neural network. The trained model 151 can be obtained, for example, by performing machine learning using training data that associates frame images with target region information and ground truth data. As ground truth data, information indicating the position of the target object 55 located within the target region in the frame image can be used. The ground truth data may further include information indicating the size of the target object 55 located within the target region in the frame image. In machine learning, the parameters of the trained model 151 are updated so that the output result of the trained model 151 approaches the ground truth data associated with the frame image input to the trained model 151.

[0053] When the target object 55 is detected, the output control unit 170 generates and outputs output data in step S106 that indicates the detection result by the first detection unit 150. The output data may include, for example, data for displaying a second target frame image 52 and a box 58 surrounding the detected target object 55 superimposed on the second target frame image 52. The output control unit 170 may output the output data to a display connected to the detection device 10 to display an image in which the second target frame image 52 and the box 58 are superimposed. By sequentially acquiring and processing multiple frame images as second target frame images 52, the first detection unit 150 can output the box 58 superimposed on the video captured by the imaging device 22. The output control unit 170 may output the output data to other devices or store it in a storage unit accessible from the output control unit 170.

[0054] In addition, the output control unit 170 may include information indicating the position and size of the target object 55 in the output data.

[0055] Since the first detection unit 150 can detect target objects 55 that newly appear in multiple time-series frame images with low latency, the detection device 10 is suitable for use in real-time monitoring, for example.

[0056] As described above, according to this embodiment, the target area setting unit 130 sets a target area in the second target frame image where processing should be performed to detect target objects not included in the first target frame image. The first detection unit 150 then executes target object detection processing on the target area within the second target frame image. Therefore, target objects 55 that newly appear in the video can be detected with low latency.

[0057] (Second Embodiment) The detection device 10 according to the second embodiment is the same as the detection device 10 according to the first embodiment, except that the target area setting unit 130 sets the target area 54 based on the attention area 56.

[0058] In the second embodiment, the target area setting unit 130 sets the target area 54 based on the interval between the imaging timing of the first target frame image 51 and the imaging timing of the second target frame image 52.

[0059] In this embodiment, the position of the object of attention 53 may change over time within the frame image, i.e., within the field of view of the imaging device 22. For example, the object of attention 53 may be a moving vehicle or the like. The imaging device 22 that captures multiple frame images may be a fixed camera or a pan / tilt camera whose imaging area changes automatically. That is, the camera may be mounted on a stage that rotates on one or two axes.

[0060] Figure 14 illustrates the relationship between the attention region 56 and the target region 54 when the position of the attention object 53 changes over time. The attention region 56 is specified with respect to the first target frame image 51. On the other hand, the second target frame image 52 is a frame image captured after the first target frame image 51, and the attention object 53 may move between the capture timing of the first target frame image 51 and the capture timing of the second target frame image 52. That is, the position and size of the target region 54 in the second target frame image 52 may not be the same as the position and size of the attention region 56 in the first target frame image 51. The target region setting unit 130 in this embodiment predicts the difference between the attention region 56 and the target region 54 and identifies the target region 54.

[0061] The target area setting unit 130 identifies multiple attention areas 56 in a time series by identifying attention areas 56 in each of the multiple first target frame images 51 from among the multiple frame images. Then, the target area setting unit 130 sets the target area 54 based on the positions of the multiple attention areas 56 that have been identified. This will be explained in detail below.

[0062] The target area setting unit 130 identifies attention areas 56 in each of the multiple first target frame images 51. In this way, multiple time-series attention areas 56 are obtained. The target area setting unit 130 also associates corresponding attention areas 56 with each other, for example, as described below.

[0063] For example, the target area setting unit 130 assigns the frame number of the first target frame image 51 and attention object identification information to the attention object information obtained by inputting the first target frame image 51 into the trained model 111. The attention object identification information is information for identifying multiple attention objects 53 in the first target frame image 51 from each other, and is, for example, an identification code. The same attention object 53 is assigned the same attention object identification information among multiple first target frame images 51 with different frame numbers. Existing techniques are used to identify the same attention object 53 among multiple first target frame images 51 with different frame numbers.

[0064] Furthermore, the target area setting unit 130 assigns attention area identification information to the attention area 56. The attention area identification information is information for identifying multiple attention areas 56 in the first target frame image 51 from each other, and is, for example, an identification code. The target area setting unit 130 assigns the same attention area identification information to attention areas 56 that are identified in different first target frame images 51 and that correspond to each other. In this way, it is possible to capture the time-series changes of the attention area 56. Coordinated attention areas 56 are attention areas 56 that have the same positional relationship with respect to attention objects 53 to which the same attention object identification information is assigned. Specifically, for example, if the same vehicle is included in multiple first target frame images 51 captured at different timings, the same attention area identification information will be assigned to the attention area 56 on the right side of the vehicle in each of the first target frame images 51.

[0065] The target area setting unit 130 identifies a target area 54 for each attention area identification piece. The target area setting unit 130 uses a plurality of time-series attention areas 56, each with the same attention area identification piece, to predict the position and size of the attention areas 56 at the time the second target frame image 52 is captured. The target area setting unit 130 may predict the position and size of the attention areas 56 by, for example, linear prediction, by a Kalman filter, or by AI (Artificial Intelligence) utilizing deep learning.

[0066] In predicting the position and size of the attention region 56 at the time the second target frame image 52 is captured, information indicating the interval between the capture timing of at least one of the processed first target frame images 51 (for example, the first target frame image 51 with the largest frame number) and the capture timing of the second target frame image 52 is further used. The information indicating the interval between the capture timing of the first target frame image 51 and the capture timing of the second target frame image 52 may be a length of time or a difference in frame numbers (for example, k in Figure 14). The interval between the capture timing of the first target frame image 51 and the capture timing of the second target frame image 52 depends on the length of processing time of the target region setting unit 130, etc., and is assumed to be known.

[0067] The target area setting unit 130 generates target area information by setting the position and size of the attention area 56 at the timing when the predicted second target frame image 52 is captured as the position and size of the target area 54. In this way, even if the position of the attention object 53 changes over time within the frame image, the area in the second target frame image 52 where there is a high probability that a new target object 55 will appear can be set as the target area 54.

[0068] Next, the operation and effects of this embodiment will be described. In this embodiment, the same operation and effects as in the first embodiment can be obtained. In addition, the target area setting unit 130 according to this embodiment sets the target area 54 based on the interval between the imaging timing of the first target frame image 51 and the imaging timing of the second target frame image 52. Therefore, even if the position of the object of attention 53 changes over time within the frame image, the target area 54 can be appropriately identified and the target object 55 can be detected with low delay.

[0069] (Modified Version) Figure 15 is a diagram illustrating the operating environment of the detection device 10. In the modified version, multiple frame images are captured by an imaging device 22 provided on the mobile body 82. The detection device 10 is connected to the imaging device 22. The target area setting unit 130 and the first detection unit 150 of the detection device 10 in the modified version acquire frame images captured by the imaging device 22 provided on the mobile body 82. In Figure 15, an example is shown where the mobile body 82 is a vehicle, but the mobile body 82 may be a drone or a person. For example, the mobile body 82 may be a wearable camera worn by a security guard or police officer.

[0070] The modified detection device 10 has the same configuration as the second embodiment. That is, even if the position of the object of attention 53 changes over time within the frame image, i.e., within the field of view of the imaging device, the target object 55 can be detected with low delay.

[0071] If the mobile object 82 is a vehicle, the detection results from the detection device 10 can be used to assist in the driving of the mobile object 82 or for autonomous driving. If the mobile object 82 is a drone or a person, the detection results from the detection device 10 can be used for surveillance, such as detecting suspicious persons.

[0072] According to this modified example, the same actions and effects as those of the second embodiment can be obtained.

[0073] (Third Embodiment) Figure 16 is a diagram illustrating the configuration of a detection device 10 equipped with a history information generation unit 160. Figure 17 is a diagram illustrating a trained model 112 used by the target area setting unit 130. The detection device 10 according to the third embodiment is the same as the detection device 10 according to the first embodiment, the second embodiment, and at least one of the modified embodiments, except for the points described below.

[0074] In this embodiment, the target region setting unit 130 identifies the attention region 56 in the first target frame image 51 by inputting the first target frame image 51 to the trained model 112. The trained model 112 is a model that detects the attention region 56 in the input frame image. Then, the target region 54 is set based on the attention region 56 in the first target frame image 51. By using the trained model 112, the attention region 56 can be detected with high accuracy.

[0075] The trained model 112 accepts a frame image as input. The trained model 112 then outputs attention region information regarding the attention region 56 contained in the input frame image. The attention region information indicates at least the position and size of the attention region 56 within the input frame image. The attention region information may also include, for example, information indicating the position and size of a box that shows the outer edge of the attention region 56 in the input frame image.

[0076] Multiple attention regions 56 may be detected for a single first target frame image 51. In the case where the target region setting unit 130 assigns attention region identification information to the attention region 56, as in the second embodiment, existing techniques are used to identify the same attention region 56 among multiple first target frame images 51 with different frame numbers.

[0077] The trained model 112 may be composed of a neural network. The trained model 112 can be obtained, for example, by performing machine learning using training data that associates frame images with ground truth data. As ground truth data, information indicating the position and size of the attention region 56 within the frame image can be used. In machine learning, the parameters of the trained model 112 are updated so that the output result of the trained model 112 approaches the ground truth data associated with the frame image input to the trained model 112.

[0078] The detection device 10 according to this embodiment may further include a second detection unit 140, a history information generation unit 160, a history storage unit 101, and a learning unit 180. The second detection unit 140 performs a process to detect a target object 55 in a first target frame image 51. The history information generation unit 160 generates history information. The history information includes the detection results of the target object 55 in a plurality of first target frame images 51. The history storage unit 101 holds the history information generated by the history information generation unit 160. The learning unit 180 performs machine learning on a trained model 112 using the history information.

[0079] The second detection unit 140 can detect the target object 55 contained in the first target frame image 51 by inputting the first target frame image 51 to a trained model that detects the target object 55 in the input frame image. Figure 18 is an example of the trained model 141 used by the second detection unit 140. The trained model 141 detects the target object 55 using the entire input frame image. The trained model 141 outputs target object information regarding the target object 55 present in the input frame image.

[0080] The trained model 141 may be composed of a neural network. The trained model 141 can be obtained, for example, by performing machine learning using training data that associates frame images with ground truth data. As ground truth data, information indicating the position of the target object 55 contained in the frame image can be used. The ground truth data may further include information indicating the size of the target object 55 contained in the frame image. In machine learning, the parameters of the trained model 141 are updated so that the output result of the trained model 141 approaches the ground truth data associated with the frame image input to the trained model 141.

[0081] The second detection unit 140 is preferably more accurate than the first detection unit 150. For example, the number of layers in the neural network of the trained model 141 is preferably greater than the number of layers in the neural network of the trained model 151. The detection of the target object 55 by the second detection unit 140 may take longer than the detection of the target object 55 by the first detection unit 150.

[0082] The history information generation unit 160 generates history information that associates the first target frame image 51 with the target object information obtained from the first target frame image 51 by the second detection unit 140. The history information generation unit 160 stores the generated history information in the history storage unit 101. In this way, the history storage unit 101 accumulates history information for multiple first target frame images 51 in a time series. Such history information includes scenes in multiple frame images (videos) in which the target object 55 newly appears. Therefore, by using the history information, a trained model 112 can be obtained that can detect regions in which there is a high probability that the target object 55 will newly appear.

[0083] The learning unit 180 reads and acquires history information for multiple time-series first target frame images 51 from the history storage unit 101. The learning unit 180 may, for example, analyze the history information to identify the first target frame image 51 and the region in the first target frame image 51 where there is a high probability that a target object 55 will newly appear, i.e., the attention region 56. The learning unit 180 may then associate the attention region information indicating the attention region 56 in the first target frame image 51 with the first target frame image 51 as ground truth data to generate training data. The learning unit 180 can then generate a trained model 112 by performing machine learning (i.e., updating the parameters of the trained model 112) using the generated training data.

[0084] The learning unit 180 may update the trained model 112 each time history information is generated, or it may update the trained model 112 at predetermined intervals.

[0085] The learning unit 180 is not limited to the example described above, and may perform machine learning on the trained model 112 using pre-prepared training data.

[0086] In addition to the target region 54, the first detection unit 150 may also perform object detection processing on regions that should be processed to detect object 55 included in the first target frame image 51 prior to the second target frame image 52. The regions that should be processed to detect object 55 included in the first target frame image 51 can be identified, for example, using the detection results of the second detection unit 140 and the technology described in Patent Document 1.

[0087] The hardware configuration of the computer implementing the detection device 10 according to this embodiment is shown, for example, in Figure 5, similar to the first embodiment. However, the storage device 1080 of the computer 1000 implementing the detection device 10 according to this embodiment further stores program modules that implement the functions of the second detection unit 140, the history information generation unit 160, and the learning unit 180 according to this embodiment. The history storage unit 101 is implemented, for example, using the storage device 1080 of the computer 1000 implementing the detection device 10.

[0088] Furthermore, the detection device 10 does not necessarily have to include a learning unit 180. The trained model 112 may be generated by machine learning performed on a device separate from the detection device 10. Also, the detection device 10 does not necessarily have to include a history storage unit 101. The history storage unit 101 may be provided outside the detection device 10.

[0089] Next, the operation and effects of this embodiment will be described. In this embodiment, the same operation and effects as in the first embodiment can be obtained. In addition, according to this embodiment, the target area setting unit 130 identifies the attention area 56 in the first target frame image 51 by inputting the first target frame image 51 to the trained model 112 that detects the attention area 56 in the input frame image. Therefore, the attention area 56 can be detected with high accuracy, and the detection accuracy of the target object 55 in the first detection unit 150 can be improved.

[0090] (Fourth Embodiment) Figure 19 is a diagram illustrating a method for setting the target area 54 using three-dimensional information. The detection device 10 according to the fourth embodiment is the same as the detection device 10 according to the first embodiment, the second embodiment, and at least one of the modified embodiments, except for the points described below.

[0091] In the fourth embodiment, the target area setting unit 130 detects discontinuous regions 70 in the three-dimensional information of the region captured by the first target frame image 51. The discontinuous regions 70 are regions where the position in the depth direction is discontinuous. The target area setting unit 130 sets the target region 54 based on the position information of the detected discontinuous regions 70 within the first target frame image 51 (hereinafter referred to as "discontinuous region information").

[0092] Figure 20 illustrates the operating environment of the detection device 10. Three-dimensional information is generated by the measuring device 72. For example, as described above in the modified example, if multiple frame images are captured by an imaging device 22 provided on the moving body 82, the measuring device 72 may be provided on the moving body 82. Examples of measuring devices 72 include LiDAR (Light Detection And Ranging), ToF (Time of Flight) sensors, and stereo cameras. The three-dimensional information generated by the measuring device 72 and the detection results of the detection device 10 may be used for driving assistance or autonomous driving of the moving body 82. In Figure 19, the right side illustrates a frame image, and the left side illustrates a three-dimensional image obtained from the three-dimensional information corresponding to that frame image. These images show a wall that is interrupted in the middle.

[0093] The measuring device 72 may be connected to the input / output interface 1100 or network interface 1120 of the computer 1000 that implements the detection device 10. The target area setting unit 130 can acquire three-dimensional information from the measuring device 72. The area captured by the measuring device 72 and the area captured by the imaging device 22 overlap in at least a portion. Preferably, the area captured by the measuring device 72 includes the entire area captured by the imaging device 22. The target area setting unit 130 acquires three-dimensional information generated at the timing when the first target frame image 51 is generated. Note that the timing when the first target frame image 51 is generated and the timing when the three-dimensional information is generated do not need to be exactly the same. The target area setting unit 130 can treat the three-dimensional information generated at the timing closest to the timing when the first target frame image 51 is generated as the three-dimensional information corresponding to the first target frame image 51.

[0094] Three-dimensional information is, for example, point cloud data containing data for multiple points whose three-dimensional coordinates have been identified. In three-dimensional information, a discontinuous region 70 is a region where the position in the depth direction is discontinuous. The depth direction is the direction away from the reference point of the three-dimensional information. An example of a reference point for three-dimensional information is the position of the measuring device 72 that generates the three-dimensional information. In the three-dimensional image, points that are farther from the reference point in the depth direction are shown in darker colors. The three-dimensional image also illustrates the distance in the depth direction from the reference point for each region.

[0095] The target area setting unit 130 analyzes the three-dimensional information to calculate the distance in the depth direction for all pairs of adjacent point data among the multiple point data. The target area setting unit 130 then identifies pairs whose calculated distance in the depth direction is greater than or equal to a threshold as discontinuous pairs. The threshold may be a predetermined fixed value, or it may be determined based on the distribution of distances calculated for all pairs (for example, based on the result of a comparison with the average value).

[0096] The target area setting unit 130 identifies a discontinuous region 70 that includes the identified discontinuous pair. The target area setting unit 130 may also identify the discontinuous region 70 such that the distance between the points included in the discontinuous pair and the outer edge of the discontinuous region 70 is greater than or equal to a predetermined distance.

[0097] Multiple discontinuous pairs may be identified. If multiple discontinuous pairs are identified, the multiple discontinuous pairs may be clustered using existing methods, and a discontinuous region 70 may be identified for each cluster.

[0098] The coordinates in the 3D information and the coordinates in the first target frame image 51 corresponding to that 3D information are known and convertible. The target area setting unit 130 converts the discontinuous region 70 into discontinuous region information that indicates the region in the first target frame image 51. The target area setting unit 130 then treats the discontinuous region information as attention region information.

[0099] Next, the operation and effects of this embodiment will be described. In this embodiment, the same operation and effects as in the first embodiment can be obtained. In addition, according to this embodiment, the target area setting unit 130 sets the target area 54 based on the positional information of the detected discontinuous area 70 within the first target frame image 51. Therefore, using three-dimensional information, it is possible to identify areas where there is a high probability that a target object 55 will newly appear, such as around the shadow of an obstruction.

[0100] (Other Embodiments) The detection device 10 according to this embodiment is the same as the detection device 10 according to at least one of the first to fourth embodiments and its modifications, except for the processing performed by the output control unit 170. In this embodiment, the output control unit 170 displays the second target frame image 52 and information indicating the target region 54 set for the second target frame image 52 overlaid on it.

[0101] In this embodiment, the output control unit 170 generates and outputs an output image as output data, for example, as shown in Figure 12. The output control unit 170 can display the output image on a display connected to the detection device 10, for example.

[0102] The output image is an image in which, for example, a box indicating the target region 54 and a box surrounding the detected target object 55 are superimposed on the second target frame image 52. In the output image, the box indicating the target region 54 and the box surrounding the target object 55 are displayed in a way that allows them to be distinguished from each other. In the example in Figure 12, the box indicating the target region 54 is shown with a dashed line, and the box surrounding the target object 55 is shown with a solid line. As another example, the box indicating the target region 54 and the box surrounding the detected target object 55 may have different colors or different line thicknesses.

[0103] Next, the operation and effects of this embodiment will be described. In this embodiment, the same operation and effects as in the first embodiment can be obtained. In addition, according to this embodiment, the output control unit 170 overlays the second target frame image 52 with information indicating the target region 54 set for the second target frame image 52. Therefore, the user of the detection device 10 can understand which region in particular was targeted for detection processing.

[0104] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0105] Furthermore, while the flowcharts used in the above description show multiple steps (processes) in sequence, the execution order of the steps performed in each embodiment is not limited to the order in which they are described. In each embodiment, the order of the illustrated steps can be changed to the extent that it does not impede the content. Also, the above embodiments can be combined to the extent that their contents do not conflict.

[0106] Some or all of the above embodiments may also be described as follows, but are not limited to the following: Appendix 1. A detection device comprising: a target area setting means for setting a target area to be processed for detecting a target object not included in the first target frame image in a second target frame image captured after the first target frame image among the multiple frame images by processing a first target frame image included in a plurality of frame images in a time series; and a first detection means for executing the target object detection process on the target area in the second target frame image. Appendix 2. A detection device according to Appendix 1, further comprising an output control means for displaying the second target frame image and information indicating the target area set for the second target frame image superimposed. Appendix 3. A detection device according to Appendix 1 or Appendix 2, further comprising an input receiving means for receiving input regarding the size of the target area. Appendix 4. A detection device according to any one of Appendix 1 to Appendix 3, wherein the target area setting means detects an object of attention contained in the first target frame image and sets the target area using the positional information of the detected object of attention within the first target frame image. Appendix 5. A detection device according to Appendix 4, wherein the target area setting means detects an object of attention contained in the first target frame image by inputting the first target frame image to a trained model that detects the object of attention in the input frame image. Appendix 6. A detection device according to Appendix 4 or Appendix 5, wherein the target area setting means further sets the target area using a label indicating the type of object of attention. Appendix 7. A detection device according to any one of Appendix 1 to Appendix 3, wherein the target area setting means identifies the object of attention in the first target frame image by inputting the first target frame image to a trained model that detects the area of ​​attention in the input frame image and sets the target area based on the area of ​​attention in the first target frame image.Note 8. A detection device according to any one of Notes 1 to 3, wherein the target area setting means detects discontinuous regions in the three-dimensional information of the region captured by the first target frame image where the position in the depth direction is discontinuous, and sets the target area based on the position information of the detected discontinuous region within the first target frame image. Note 9. A detection device according to any one of Notes 1 to 8, wherein the target area setting means sets the target area based on the interval between the imaging timing of the first target frame image and the imaging timing of the second target frame image. Note 10. A detection device according to Note 9, wherein the target area setting means identifies a plurality of attention regions in a time series by identifying attention regions in each of the plurality of first target frame images among the plurality of frame images, and sets the target area based on the positions of the identified plurality of attention regions. Note 11. A detection device according to any one of Notes 1 to 10, wherein the target area setting means includes the edge of the second target frame image in the target area. Note 12. A detection device according to any one of Notes 1 to 11, further comprising: a second detection means for performing a process to detect the target object in the first target frame image; and a history information generation means for generating history information including the detection results of the target object in a plurality of the first target frame images. Note 13. A detection device according to Note 12, wherein the second detection means is more accurate than the first detection means. Note 14. A detection device according to any one of Notes 1 to 13, wherein the target object is a moving object. Note 15. A detection method in which one or more computers process a first target frame image included in a plurality of frame images in a time series to set a target region in a second target frame image captured after the first target frame image among the plurality of frame images, in which a process should be performed to detect a target object not included in the first target frame image; and perform the target object detection process in the target region within the second target frame image.Note 16. A program that causes a computer to function as a target area setting means for setting a target area in a second target frame image captured after the first target frame image among the multiple frame images, by processing a first target frame image included in a plurality of frame images in a time series, in order to perform processing for detecting a target object not included in the first target frame image, and as a first detection means for performing the target object detection processing on the target area in the second target frame image. Note 17. A computer-readable recording medium that stores a program, wherein the program is a recording medium that causes a computer to function as a target area setting means for setting a target area in a second target frame image captured after the first target frame image among the plurality of frame images, by processing a first target frame image included in a plurality of frame images in a time series, in order to perform processing for detecting a target object not included in the first target frame image, and as a first detection means for performing the target object detection processing on the target area in the second target frame image. Note 18. A system comprising a detection device and an imaging device that generates a plurality of frame images in a time series, wherein the detection device processes a first target frame image included in the plurality of frame images to set a target region in a second target frame image captured after the first target frame image among the plurality of frame images, for which processing should be performed to detect a target object not included in the first target frame image, and a first detection means that executes the target object detection process on the target region in the second target frame image. Appendix 19. A detection method comprising generating a plurality of frame images in a time series with an imaging device, processing a first target frame image included in the plurality of frame images to set a target region in a second target frame image captured after the first target frame image among the plurality of frame images, for which processing should be performed to detect a target object not included in the first target frame image, and executing the target object detection process on the target region in the second target frame image.

[0107] Furthermore, some or all of the configurations described in Appendices 2 to 14, which are subordinate to Appendice 1 above, may also be subordinate to each of Appendices 15 to 19 in the same way as Appendices 2 to 14. Moreover, not limited to Appendice 1 and Appendices 15 to 19, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0108] This application claims priority based on Japanese Patent Application No. 2024-200567, filed on 18 November 2024, and incorporates all of its disclosures herein.

[0109] 10 Detection device 20 System 22 Imaging device 51 First target frame image 52 Second target frame image 53 Object of attention 54 Target area 55 Target object 56 Area of ​​attention 57 Edge 58 Box 59 Outer edge 70 Discontinuous area 72 Measurement device 82 Moving object 101 History storage unit 111 Trained model 112 Trained model 120 Input reception unit 130 Target area setting unit 140 Second detection unit 141 Trained model 150 First detection unit 151 Trained model 160 History information generation unit 170 Output control unit 180 Learning unit 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface

Claims

1. A detection device comprising: a target area setting means for setting a target area in a second target frame image captured after the first target frame image among the multiple frame images, by processing a first target frame image included in a plurality of time-series frame images, in which processing should be performed to detect a target object not included in the first target frame image; and a first detection means for executing the target object detection process on the target area in the second target frame image.

2. A detection device according to claim 1, further comprising output control means for displaying the second target frame image and information indicating the target region set for the second target frame image in superimposed form.

3. A detection device according to claim 1 or 2, further comprising an input receiving means for receiving input regarding the size of the target area.

4. A detection device according to any one of claims 1 to 3, wherein the target area setting means detects a attention object included in the first target frame image and sets the target area using the positional information of the detected attention object within the first target frame image.

5. A detection device according to claim 4, wherein the target area setting means inputs the first target frame image to a trained model that detects the attention object in the input frame image, thereby detecting the attention object included in the first target frame image.

6. A detection device according to claim 4 or 5, wherein the target area setting means further uses a label indicating the type of object to be noticed to set the target area.

7. A detection device according to any one of claims 1 to 3, wherein the target area setting means inputs the first target frame image to a trained model that detects attention areas in an input frame image, thereby identifying the attention area in the first target frame image, and setting the target area based on the attention area in the first target frame image.

8. A detection device according to any one of claims 1 to 3, wherein the target area setting means detects discontinuous regions in which the position in the depth direction is discontinuous in the three-dimensional information of the region captured in the first target frame image, and sets the target region based on the position information of the detected discontinuous region in the first target frame image.

9. A detection device according to any one of claims 1 to 8, wherein the target area setting means sets the target area based on the interval between the imaging timing of the first target frame image and the imaging timing of the second target frame image.

10. A detection device according to claim 9, wherein the target area setting means identifies a plurality of attention areas in a time series by identifying a plurality of attention areas in each of the plurality of first target frame images among the plurality of frame images, and sets the target area based on the positions of the plurality of attention areas identified.

11. A detection device according to any one of claims 1 to 10, wherein the target area setting means includes the edge of the second target frame image in the target area.

12. A detection device according to any one of claims 1 to 11, further comprising: a second detection means for performing a process to detect the target object in the first target frame image; and a history information generation means for generating history information including the detection results of the target object in a plurality of the first target frame images.

13. A detection device according to claim 12, wherein the second detection means is more accurate than the first detection means.

14. A detection device according to any one of claims 1 to 13, wherein the target object is a moving object.

15. A detection method comprising one or more computers processing a first target frame image included in a plurality of time-series frame images to set a target region in a second target frame image captured after the first target frame image among the plurality of frame images, in which processing should be performed to detect a target object not included in the first target frame image, and executing the target object detection processing on the target region in the second target frame image.

16. A computer-readable recording medium that stores a program, wherein the program causes the computer to function as a target area setting means for setting a target area in a second target frame image captured after the first target frame image among the multiple frame images, by processing a first target frame image included in a plurality of frame images in a time series, for which processing should be performed to detect a target object not included in the first target frame image, and as a first detection means for performing the target object detection processing on the target area in the second target frame image.

Images