Monitoring system, monitoring method, and monitoring program

Abstract

In the present invention, an object position calculation unit calculates the position of an object detected by a fixed-point sensor which is installed at a fixed position. A bird's-eye view creation unit creates, from an image captured with a moving camera mounted on a moving object, a bird's-eye view in which the capturing direction of the moving camera is viewed from above. An integration unit generates an integrated image which is an image obtained by integrating the detected object in a corresponding position on the bird's eye view. An output unit outputs the integrated image.

Description

Monitoring System, Monitoring Method, and Monitoring Program

[0001] The present disclosure relates to a monitoring system, a monitoring method, and a monitoring program for monitoring an object using a plurality of pieces of information.

[0002] In recent years, the technology of automatic driving of vehicles has been actively developed. When driving a vehicle automatically, it is required to monitor the situation around the vehicle in consideration of safety.

[0003] For example, Patent Document 1 describes a device for improving the monitoring efficiency when remotely monitoring and controlling a moving object. The device described in Patent Document 1 displays, on a monitoring screen, video data acquired by a camera that captures the outside direction from a vehicle and video data (images) obtained by processing the video data around the vehicle for each of a plurality of vehicles.

[0004] Japanese Patent No. 7290199

[0005] In remote monitoring, as the amount of video data captured by a plurality of cameras increases, a large amount of information regarding the monitoring target can be acquired. On the other hand, since the monitor needs to monitor each video displayed on a plurality of screens, the burden on the monitor increases. Therefore, it is preferable that the monitoring efficiency can be improved even when performing remote monitoring using a plurality of pieces of information.

[0006] Therefore, an object of the present disclosure is to provide a monitoring system, a monitoring method, and a monitoring program that can improve the monitoring efficiency when performing remote monitoring using a plurality of pieces of information.

[0007] The monitoring system according to the present disclosure includes an object position calculation unit that calculates the position of an object detected by a fixed-point sensor, which is a sensor installed at a fixed position, a bird's-eye view creation unit that creates a bird's-eye view of the shooting direction of a moving camera, which is a camera mounted on a moving object, from an image captured by the moving camera as seen from above, an integration unit that generates an integrated image, which is an image in which the detected object is integrated at the corresponding position in the bird's-eye view, and an output unit that outputs the integrated image.

[0008] The monitoring method described herein is characterized by calculating the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; creating a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the shooting direction of the mobile camera from above; generating an integrated image, which is an image in which the detected object is integrated into the corresponding position on the bird's-eye view; and outputting the integrated image.

[0009] The monitoring program described herein is characterized by causing a computer to perform the following: an object position calculation process that calculates the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; a bird's-eye view creation process that creates a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the direction in which the mobile camera is shooting from above; an integration process that generates an integrated image, which is an image in which the detected objects are integrated into the corresponding positions on the bird's-eye view; and an output process that outputs the integrated image.

[0010] According to this disclosure, monitoring efficiency can be improved when remote monitoring is performed using multiple pieces of information.

[0011] This is a block diagram showing an example configuration of one embodiment of the monitoring system according to this disclosure. This is an explanatory diagram showing an example of image merging process. This is an explanatory diagram showing an example of image merging process when an obstacle is present. This is a flowchart showing an example of the operation of the monitoring system. This is a block diagram showing a modified version of the monitoring system according to this disclosure. This is a block diagram showing an example configuration of another embodiment of the monitoring system according to this disclosure. This is a flowchart showing another example of the operation of the monitoring system of this embodiment. This is a block diagram showing an overview of the monitoring system according to this disclosure. This is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] Embodiment 1. Figure 1 is a block diagram showing an example configuration of a first embodiment of the monitoring system according to the present disclosure. The monitoring system 100 of this embodiment includes a storage unit 10, an input unit 20, an object recognition unit 30, a bird's-eye view creation unit 40, an integration unit 50, an output unit 60, and a display device 70. The display device 70 is implemented, for example, by a display device.

[0014] Furthermore, the monitoring system 100 of this embodiment is connected to a mobile body 200 equipped with a camera 210 and a fixed-point camera 300 in a communicative manner. The mobile body 200 assumed in this embodiment is a mobile body that is the target of remote monitoring, and an example of this is an autonomous vehicle. However, the mobile body is not limited to an autonomous vehicle. The mobile body 200 may be, for example, a general vehicle or bus, or a mobile robot such as a delivery robot or a drone.

[0015] Furthermore, the camera 210 can be implemented as, for example, an in-vehicle camera. The control device 220 acquires the image or video (hereinafter simply referred to as "image") captured by the camera 210 and transmits it to the monitoring system 100 along with the identification information of the camera 210. In addition, the control device 220 acquires the time, location information, and orientation (direction of movement) when the camera 210 captured the image and transmits them to the monitoring system 100 along with the image.

[0016] The method by which the control device 220 acquires position information and direction is arbitrary. The control device 220 may acquire position information, for example, via a GPS (Global Positioning System) receiver (not shown) provided by the moving object. Alternatively, the control unit 220 may acquire direction, for example, via a GPS / CAN (Controller Area Network) used by the moving object 200.

[0017] Furthermore, the fixed-point camera 300 assumed in this embodiment is installed at a fixed position, such as beside the road on which the moving object 200 travels, and captures images of objects present around the road. The fixed-point camera 300 is a device equipped with an image sensor, and can be described as a type of sensor installed at a fixed position (hereinafter referred to as a fixed-point sensor). However, the sensor is not limited to a camera, and may be, for example, a sensor used in LiDAR (Light Detection and Ranging). In this embodiment, a camera is used as an example of a sensor, and the position of an object is determined by capturing an image with the image sensor mounted on the fixed-point camera 300.

[0018] The fixed-point camera 300 is an infrastructure camera installed, for example, to monitor the condition of infrastructure such as roads. Preferably, the fixed-point camera 300 is installed in a location where it is difficult to capture images with the camera 210 installed on the mobile unit 200.

[0019] Examples of objects that the fixed-point camera 300 will capture include, for example, traffic participants such as pedestrians, cyclists, motorcyclists, and car drivers. However, the objects captured by the fixed-point camera 300 are not limited to traffic participants. In addition, any object may be captured in the images taken by the fixed-point camera 300.

[0020] The fixed-point camera 300 transmits the captured image to the monitoring system 100 along with the identification information of the fixed-point camera 300. Furthermore, the fixed-point camera 300 obtains the time the image was captured and transmits it to the monitoring system 100 along with the image. If the imaging range of the fixed-point camera 300 changes (for example, the field of view changes), the fixed-point camera 300 may transmit information indicating the field of view at the time the image was captured to the monitoring system 100.

[0021] In Figure 1, an example is shown where there is one mobile unit 200 (more specifically, one camera 210) and one fixed camera 300. However, the number of mobile units 200 (more specifically, cameras 210) and fixed cameras 300 is not limited to one; there may be two or more.

[0022] The memory unit 10 stores various types of information that the monitoring system 100 uses for processing. In this embodiment, the memory unit 10 stores information about the camera 210 and information about the fixed-point camera 300, associating them with the camera identification information. As information about the camera 210, the memory unit 10 stores, for example, the installation position (height from the ground) on the mobile unit 200.

[0023] Furthermore, the memory unit 10 stores information related to the fixed-point camera 300, such as the installation location (latitude and longitude, height from the ground), installation angle, and field of view. The memory unit 10 may also store information other than that mentioned above regarding the camera. The memory unit 10 is implemented, for example, by a magnetic disk.

[0024] The input unit 20 receives input of various information transmitted from the camera 210 (more specifically, the control device 220) and the fixed-point camera 300. In addition, the input unit 20 may also receive operation instructions for the display device 70 from the observer.

[0025] The object recognition unit 30 performs object recognition on the image transmitted from the fixed-point camera 300 (hereinafter referred to as the first image). The type of object that the object recognition unit 30 performs object recognition on is arbitrary and predetermined. For example, the object recognition unit 30 may perform object recognition on traffic participants. Since the method of performing object recognition from images is widely known, a detailed explanation is omitted here. Also, the number of objects that the object recognition unit 30 performs object recognition on is not limited to one, but may be two or more.

[0026] Next, the object recognition unit 30 calculates the actual location of the recognized object. The actual location is expressed, for example, in terms of latitude and longitude. The method by which the object recognition unit 30 calculates the actual location is arbitrary. For example, if the area captured by the fixed-point camera 300 is fixed, it is possible to pre-associate each pixel of the captured image with the actual location based on the installation conditions such as the installation position (latitude and longitude, height from the ground), installation angle, and field of view. In that case, the object recognition unit 30 may identify the corresponding actual location from the area of ​​the recognized object in the first image.

[0027] Furthermore, if the imaging range of the fixed-point camera 300 changes, the object recognition unit 30 may determine the actual location of the recognized object by calculating the distance from the fixed-point camera 300's installation position (height from the ground), installation angle, and field of view. Since methods for calculating the distance between the camera and an object in the image based on the camera's installation status are widely known, a detailed explanation is omitted here.

[0028] The bird's-eye view creation unit 40 creates a bird's-eye view from the image captured by the camera 210 (hereinafter referred to as the second image), showing the direction in which the camera 210 is shooting, as viewed from above. Here, since the position and orientation are also acquired when the second image is captured, the bird's-eye view creation unit 40 may create the bird's-eye view from the captured second image, associating it with the position and orientation in which the second image was captured.

[0029] For example, if camera 210 is an in-vehicle camera installed in front of the vehicle that captures images of the direction of travel, the bird's-eye view creation unit 40 may create a bird's-eye view of the vehicle's direction of travel, as seen from above, in correspondence with the vehicle's position information (latitude and longitude) and bearing (direction of travel). Note that the direction captured by camera 210 is not limited to the direction of travel, but may also be the rear.

[0030] The method by which the bird's-eye view creation unit 40 creates the bird's-eye view is arbitrary. For example, the bird's-eye view creation unit 40 may estimate the angle of the ground from the installation position of the camera 210 and a point at infinity, and convert it into an image (bird's-eye view) of the direction of travel (for example, a road) viewed from above. Note that methods for generating a bird's-eye view from an image are also widely known, so a detailed explanation is omitted here.

[0031] The integration unit 50 generates an image in which the objects recognized by the object recognition unit 30 are integrated into the corresponding positions on the bird's-eye view map created by the bird's-eye view map creation unit 40. Specifically, the integration unit 50 calculates the actual position indicated by each pixel on the created bird's-eye view map based on the position (latitude and longitude) and orientation of the camera 210. Then, the integration unit 50 generates an image (hereinafter sometimes referred to as the integrated image) in which the objects recognized by the object recognition unit 30 are integrated into the same positions on the bird's-eye view map.

[0032] Furthermore, when camera 210 and fixed-point camera 300 transmit continuous images (i.e., video), the integration unit 50 only needs to perform integration processing on images captured at the same time.

[0033] The integration unit 50 then generates an image by performing a transformation on the integrated image that returns it to the image before the bird's-eye view was generated. In other words, the integration unit 50 transforms the integrated image into an image from the viewpoint of the camera 210. This generates an image in which the objects recognized by the object recognition unit 30 are integrated into the second image. The integration unit 50 can then perform a transformation (inverse transformation) to return the bird's-eye view to the original image using a known method.

[0034] Furthermore, the method by which the integration unit 50 integrates objects into an image is not limited to integrating objects into a bird's-eye view. The integration unit 50 may first identify the coordinates of the bird's-eye view corresponding to the position of the detected object, after performing a transformation back to the original image (the image before the bird's-eye view was generated). Then, the integration unit 50 may integrate the detected object at the position of the identified coordinates in the original image. In this case, the integration unit 50 may integrate the detected object itself, or it may integrate an icon or illustration representing that object. In this case, the integration unit 50 does not need to perform an inverse transformation on the bird's-eye view itself.

[0035] Figure 2 is an explanatory diagram illustrating an example of the image integration process. First, the fixed-point camera 300 transmits an image 12 of object 11 to the monitoring system 100. Meanwhile, the control device 220 transmits an image 13 captured by camera 210 to the monitoring system 100. The object recognition unit 30 recognizes object 11 from image 12 and calculates the actual position of the recognized object. The bird's-eye view creation unit 40 creates a bird's-eye view 14 from image 13.

[0036] Here, it is determined that the position of object 11 corresponds to the position indicated by star 15 in the bird's-eye view 14. In this case, the integration unit 50 creates an integrated image in which object 11 is integrated to the position indicated by star 15, and generates an image that is converted back to the image before the bird's-eye view was generated. Then, the output unit 60 outputs the generated image 16 to the display device 70.

[0037] Furthermore, to improve safety, fixed-point cameras 300 are often installed in locations with poor visibility. In such situations, objects recognized by the fixed-point camera 300 may be hidden behind obstacles such as buildings and located in a blind spot for the mobile device 200. If conversion is performed in this state, there is a risk that the recognized objects will be hidden behind obstacles and will not be displayed in the image.

[0038] Therefore, if the object to be integrated is hidden by an obstacle captured by the camera 210 (i.e., becomes invisible to the moving body 200), the integration unit 50 may generate an image in which the object hidden by the obstacle is displayed in front of the obstacle, and the part of the object hidden by the obstacle is displayed in a different manner from the other parts (i.e., the part visible to the moving body 200). The integration unit 50 may, for example, generate an image in which the color of the hidden part is changed, or generate an image in which the hidden object is displayed as an illustration or figure. Furthermore, if the object may be hidden by multiple obstacles, the integration unit 50 may generate an image in which the density of the object is displayed in stages according to the number of obstacles that are hiding it.

[0039] Figure 3 is an explanatory diagram illustrating an example of image integration when obstacles are present. In the example shown in Figure 3, camera 210 captures image 21. Meanwhile, obstacles 22 and 23 are present in front of the vehicle in the direction of travel, and a fixed-point camera 300 is capturing an image of the area behind obstacle 23. Furthermore, it is assumed that object 27 is present behind obstacle 23.

[0040] At this time, the object recognition unit 30 recognizes the object 27 from the image captured by the fixed-point camera 300 and calculates the actual position of the recognized object. The bird's-eye view creation unit 40 creates a bird's-eye view 24 from the image 21.

[0041] Here, it is assumed that the position of the object 27 is specified to correspond to the position indicated by the star 25 in the bird's-eye view 24. In this case, the integration unit 50 creates an integrated image in which the object 27 is integrated at the position indicated by the star 15. At this time, the integration unit 50 displays the object 27, which is hidden by the obstacle 22, on the front side of the obstacle 22, and generates an image 26 that displays the portion of the object 27 hidden by the obstacle 22 in a different manner. In FIG. 3, an example in which the object 27 is displayed on the front side of the obstacle 22 in a lighter color than other vehicles is shown.

[0042] Note that, in the example shown in FIG. 3, the case where the object 27 is completely hidden by the obstacle 22 is illustrated. For example, when the object 27 is partially hidden by the obstacle 22, the integration unit 50 may generate an image that displays the portion of the object 27 hidden by the obstacle 22 in a different manner from other portions of the object 27.

[0043] The output unit 60 outputs an image in which the object recognized by the object recognition unit 30 is integrated at the corresponding position in the bird's-eye view to the display device 70. Specifically, the output unit 60 outputs an image obtained by performing a conversion to return the integrated image to the image before the bird's-eye view is generated to the display device 70.

[0044] The input unit 20, the object recognition unit 30, the bird's-eye view creation unit 40, the integration unit 50, and the output unit 60 are realized by a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) of a computer that operates according to a program (monitoring program).

[0045] For example, the program is stored in the storage unit 10 of the monitoring system 100, and the processor may read the program and operate as the input unit 20, the object recognition unit 30, the bird's-eye view creation unit 40, the integration unit 50, and the output unit 60 according to the program. Also, the functions of the monitoring system 100 may be provided in the form of SaaS (Software as a Service).

[0046] Further, the input unit 20, the object recognition unit 30, the bird's-eye view creation unit 40, the integration unit 50, and the output unit 60 may each be implemented by dedicated hardware. Also, some or all of the components of each device may be implemented by general-purpose or dedicated circuitry, processors, etc., or combinations thereof. These may be constituted by a single chip or by a plurality of chips connected via a bus. Some or all of the components of each device may be implemented by a combination of the above-described circuitry, etc. and a program.

[0047] Further, when some or all of the components of the monitoring system 100 are implemented by a plurality of information processing devices, circuitry, etc., the plurality of information processing devices, circuitry, etc. may be centrally arranged or distributed. For example, the information processing devices, circuitry, etc. may be implemented in a form in which each is connected via a communication network, such as a client-server system, a cloud computing system, etc.

[0048] Next, the operation of the monitoring system 100 of the present embodiment will be described. FIG. 4 is a flowchart showing an operation example of the monitoring system 100 of the present embodiment. The object recognition unit 30 calculates the position of the object imaged by the fixed-point camera 300 (step S11). The bird's-eye view creation unit 40 creates a bird's-eye view from the image captured by the camera 210 of the moving body 200 (step S12). The integration unit 50 generates an integrated image in which the captured object is integrated at the corresponding position in the bird's-eye view (step S13). Then, the output unit 60 outputs the integrated image (step S14).

[0049] As described above, in this embodiment, the object recognition unit 30 calculates the position of an object detected by a fixed-point sensor, such as a fixed-point camera 300, and the bird's-eye view creation unit 40 creates a bird's-eye view from the image captured by the camera 210 mounted on the mobile body 200, showing the camera's shooting direction as seen from above. Then, the integration unit 50 generates an integrated image by integrating the detected objects into their corresponding positions on the bird's-eye view, and the output unit 60 outputs the integrated image. Therefore, the monitoring efficiency when performing remote monitoring using multiple pieces of information can be improved. In other words, using the monitoring system disclosed herein reduces the workload on remote monitoring operators and contributes to safety by eliminating the need to constantly monitor multiple images.

[0050] Next, a modified version of the monitoring system 100 of this embodiment will be described. In the above embodiment, a fixed-point camera 300 was used as an example of a fixed-point sensor. In this modified version, a case in which the fixed-point sensor is implemented using a sensor used in LiDAR will be described. A sensor used in LiDAR is, for example, a combination of a laser and an optical sensor that detects that laser.

[0051] Figure 5 is a block diagram showing a modified example of the first embodiment of the monitoring system according to the present disclosure. The monitoring system 100a of this modified example includes a storage unit 10, an input unit 21, an object recognition unit 31, a bird's-eye view creation unit 40, an integration unit 50, an output unit 60, and a display device 70. The contents of the storage unit 10, the bird's-eye view creation unit 40, the integration unit 50, the output unit 60, and the display device 70 are the same as in the first embodiment.

[0052] Furthermore, the monitoring system 100a of this modified example is communicatively connected to a control device 320 equipped with a sensor 310 (for example, an optical sensor) used in LiDAR as a fixed-point sensor.

[0053] The control device 320 acquires information about the object detected by the sensor 310 and transmits it to the monitoring system 100a along with the identification information of the sensor 310 or the control device 320. Alternatively, the control device 320 may perform a process to recognize the object from the information detected by the sensor 310 and transmit the recognition result to the monitoring system 100a. Since methods for detecting objects using LiDAR and methods for recognizing objects from the detected information are widely known, a detailed explanation is omitted here.

[0054] The input unit 21 receives various types of information transmitted from the control device 320.

[0055] The object recognition unit 31 calculates the position of the object detected by the sensor 310 based on various information received from the control device 320. Specifically, the object recognition unit 31 calculates the actual position of the object based on the distance to the object detected by the sensor 310, its shape, and other factors.

[0056] The subsequent processing is the same as in the first embodiment. That is, the integration unit 50 generates an integrated image by integrating the detected objects into their corresponding positions on the bird's-eye view. The input unit 21, object recognition unit 31, bird's-eye view creation unit 40, integration unit 50, and output unit 60 are all implemented by a computer processor that operates according to a program (monitoring program).

[0057] This configuration also improves monitoring efficiency when performing remote monitoring using multiple sources of information.

[0058] Embodiment 2. Next, a second embodiment of the monitoring system of the present disclosure will be described. In the first embodiment, a new image was created by integrating images captured by a camera mounted on a mobile body with objects detected by a fixed camera or sensor. On the other hand, since objects in overlapping areas are detected by other cameras or sensors, it is assumed that the same object will be detected multiple times. Therefore, in this embodiment, a method for generating an image by excluding overlapping objects during integration will be described.

[0059] Figure 6 is a block diagram showing an example configuration of a second embodiment of the monitoring system according to the present disclosure. The monitoring system 200 of this embodiment includes a storage unit 10, an input unit 20, an object recognition unit 30, a bird's-eye view creation unit 41, an integration unit 51, an output unit 60, and a display device 70.

[0060] The contents of the storage unit 10, input unit 20, object recognition unit 30, output unit 60, and display device 70 are the same as in the first embodiment. In this embodiment, a method for detecting objects using a fixed-point camera is described. However, as shown in the modified example of the first embodiment, objects may be detected using LiDAR.

[0061] The bird's-eye view creation unit 41 creates a bird's-eye view from the second image, similar to the bird's-eye view creation unit 40 in the first embodiment. Furthermore, the bird's-eye view creation unit 41 calculates the positions of objects in the image captured by the camera 210. Specifically, the bird's-eye view creation unit 41 performs object recognition in the second image and calculates the actual positions of the recognized objects.

[0062] The bird's-eye view creation unit 41 may calculate the actual position of an object from a second image before creating the bird's-eye view, or it may calculate the actual position of an object from the bird's-eye view. The method for calculating the actual position of an object from the second image may be the same as the method used by the object recognition unit 30 to calculate the position from the first image. The method for calculating the actual position of an object from the bird's-eye view may be the same as the method used by the integration unit 50 in the first embodiment to calculate the actual position indicated by each pixel in the bird's-eye view.

[0063] The integration unit 51, similar to the integration unit 50 in the first embodiment, generates an image in which the objects recognized in the first image are integrated into the corresponding positions in the bird's-eye view. In doing so, the integration unit 51 excludes the integration of objects that are located in the same position as the objects identified in the bird's-eye view among the objects recognized in the first image.

[0064] Furthermore, if the type and shape of an object are detected from each image, the integration unit 51 may exclude objects located at the same position from integration if their type and shape satisfy predetermined similarity criteria. This improves the accuracy of the integration.

[0065] The input unit 20, object recognition unit 30, bird's-eye view creation unit 41, integration unit 51, and output unit 60 are all implemented by a computer processor that operates according to a program (monitoring program).

[0066] Next, the operation of the monitoring system 200 of this embodiment will be described. Figure 7 is a flowchart showing an example of the operation of the monitoring system 200 of this embodiment. The process from calculating the position of the object captured by the fixed-point camera 300 to creating a bird's-eye view from the image captured by the camera 210 is the same as steps S11 and S12 illustrated in Figure 4.

[0067] The bird's-eye view creation unit 41 calculates the positions of objects in the image captured by the camera 210 of the mobile unit 200 (step S21). The integration unit 51 excludes the integration of objects that are located at the same position as the objects identified in the bird's-eye view among the objects captured by the fixed-point camera 300, and generates an integrated image integrated at the corresponding positions in the bird's-eye view (step S22). The subsequent process of outputting the integrated image is the same as step S14 illustrated in Figure 4.

[0068] As described above, in this embodiment, the bird's-eye view creation unit 41 calculates the positions of objects in the image captured by the camera 210, and the integration unit 51 excludes the integration of objects that are located at the same position as the objects identified in the bird's-eye view, among the objects detected by the fixed-point sensor, such as the fixed-point camera 300. Therefore, in addition to the effects of the first embodiment, it is possible to suppress the display of unnecessary objects.

[0069] Next, an overview of the present disclosure will be described. Figure 8 is a block diagram illustrating the overview of the monitoring system according to the present disclosure. The monitoring system 80 according to the present disclosure (for example, monitoring systems 100, 100a, 200) includes an object position calculation unit 81 (for example, object recognition units 30, 31) that calculates the position of an object detected by a fixed-point sensor (for example, a fixed-point camera 300, LiDAR) which is a sensor installed at a fixed position; a bird's-eye view creation unit 82 (for example, bird's-eye view creation units 40, 41) that creates a bird's-eye view from an aerial view of the shooting direction of a mobile camera (for example, a camera 210) mounted on a mobile body (for example, a mobile body 200) based on an image captured by the mobile camera; an integration unit 83 (for example, integration units 50, 51) that generates an integrated image which is an image in which the detected objects are integrated to corresponding positions (for example, the same position) in the bird's-eye view; and an output unit 84 (for example, output unit 60) that outputs the integrated image.

[0070] Such a configuration can improve monitoring efficiency when performing remote monitoring using multiple sources of information.

[0071] Furthermore, the output unit 84 may output an image in which the integrated image has been transformed back to the image before the bird's-eye view was generated. With this configuration, monitoring can be performed in a manner similar to the field of view of a moving object.

[0072] Furthermore, the integration unit 83 may calculate the actual location indicated by each pixel of the created bird's-eye view map based on the position (e.g., latitude and longitude) and orientation of the moving camera, and integrate the detected objects into the corresponding positions on the bird's-eye view map.

[0073] Specifically, the integration unit 83 may generate an image by converting the integrated image back to the image before the bird's-eye view was generated, and the output unit 84 may output the generated image.

[0074] Furthermore, the integration unit 83 may generate an image in which an object that is hidden by an obstacle, as captured by the moving camera, is displayed in front of the obstacle, and the part of the object that is hidden by the obstacle is displayed in a different manner from the rest of the object.

[0075] Alternatively, the object position calculation unit 81 may calculate the position of an object in an image captured by a fixed-point camera (for example, a fixed-point camera 300) which is a camera set at a fixed position.

[0076] Alternatively, the object position calculation unit 81 may calculate the position of an object detected by LiDAR from a fixed position.

[0077] Furthermore, the bird's-eye view creation unit 82 (for example, the bird's-eye view creation unit 41) calculates the positions of objects in the images captured by the mobile camera, and the integration unit 83 (for example, the integration unit 51) may exclude the integration of objects detected by the fixed-point sensor that are located at the same position as objects identified in the bird's-eye view.

[0078] Figure 9 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 1000 comprises a processor 1001, a main memory 1002, an auxiliary memory 1003, and an interface 1004.

[0079] The aforementioned monitoring system 80 is implemented in the computer 1000. The operation of each processing unit described above is stored in the auxiliary storage device 1003 in the form of a program (monitoring program). The processor 1001 reads the program from the auxiliary storage device 1003, loads it into the main memory 1002, and executes the above processing according to the program.

[0080] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-temporary tangible medium. Other examples of non-temporary tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memory), DVD-ROMs (Read-only memory), and semiconductor memory connected via the interface 1004. Furthermore, if this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may expand it into the main memory 1002 and execute the above processing.

[0081] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in the auxiliary storage device 1003.

[0082] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0083] (Note 1) A surveillance system characterized by comprising: an object position calculation unit that calculates the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; a bird's-eye view creation unit that creates a bird's-eye view of the shooting direction of the mobile camera from images captured by a mobile camera, which is a camera mounted on a moving body, as viewed from above; an integration unit that generates an integrated image, which is an image in which the detected objects are integrated into the corresponding positions on the bird's-eye view; and an output unit that outputs the integrated image.

[0084] (Note 2) The output unit is the monitoring system described in Note 1, which outputs an image in which the integrated image has been converted back to the image before the bird's-eye view was generated.

[0085] (Note 3) The monitoring system according to Note 1 or Note 2, wherein the integration unit calculates the actual position indicated by each pixel of the created bird's-eye view based on the position and orientation of the mobile camera, and integrates the detected object to the corresponding position on the bird's-eye view.

[0086] (Note 4) The monitoring system described in any one of Notes 1 to 3, wherein the integration unit generates an image by converting the integrated image back to the image before the bird's-eye view was generated, and the output unit outputs the generated image.

[0087] (Note 5) The monitoring system according to Note 1, wherein the integration unit identifies the coordinates obtained when the coordinates of the bird's-eye view corresponding to the position of the detected object are transformed back to the image before the bird's-eye view was generated, and integrates the detected object at the position of the identified coordinates in the image.

[0088] (Note 6) The monitoring system according to any one of Notes 1 to 5, wherein the integration unit displays an object that is obscured by an obstacle, as captured by a mobile camera, on the side in front of the obstacle, and generates an image in which the part of the object that is obscured by the obstacle is displayed in a different manner from the other parts of the object.

[0089] (Note 7) The object position calculation unit calculates the position of an object in an image captured by a fixed-point camera, which is a camera set at a fixed position, according to any one of Notes 1 to 6 of the monitoring system.

[0090] (Note 8) The object position calculation unit is a monitoring system according to any one of Notes 1 to 6 that calculates the position of an object detected by LiDAR (Light Detection And Ranging) from a fixed position.

[0091] (Note 9) The monitoring system described in any one of Notes 1 to 8, wherein the bird's-eye view creation unit calculates the position of objects in images captured by a mobile camera, and the integration unit excludes the integration of objects that are located at the same position as objects identified in the bird's-eye view among the objects detected by the fixed-point sensor.

[0092] (Note 10) A monitoring method characterized by calculating the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; creating a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the shooting direction of the mobile camera from above; generating an integrated image, which is an image in which the detected object is integrated into the corresponding position on the bird's-eye view; and outputting the integrated image.

[0093] (Note 11) A storage medium for storing a monitoring program that causes a computer to perform the following processes: an object position calculation process for calculating the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; a bird's-eye view creation process for creating a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the shooting direction of the mobile camera from above; an integration process for generating an integrated image, which is an image in which the detected objects are integrated into the corresponding positions on the bird's-eye view; and an output process for outputting the integrated image.

[0094] (Note 12) A monitoring program that causes a computer to perform the following: object position calculation processing, which calculates the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; bird's-eye view creation processing, which creates a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the shooting direction of the mobile camera from above; integration processing, which generates an integrated image, which is an image in which the detected objects are integrated into the corresponding positions on the bird's-eye view; and output processing, which outputs the integrated image.

[0095] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0096] 10 Memory Unit 20, 21 Input Unit 30, 31 Object Recognition Unit 40, 41 Bird's-eye View Creation Unit 50, 51 Integration Unit 60 Output Unit 70 Display Device 100, 100a, 200 Monitoring System 200 Mobile Unit 210 Camera 220 Control Device 300 Fixed Camera 310 Sensor 320 Control Device

Claims

1. A surveillance system comprising: an object position calculation unit that calculates the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; a bird's-eye view creation unit that creates a bird's-eye view of the shooting direction of the mobile camera from images captured by a mobile camera, which is a camera mounted on a moving body, as viewed from above; an integration unit that generates an integrated image, which is an image in which the detected objects are integrated into the corresponding positions on the bird's-eye view; and an output unit that outputs the integrated image.

2. The monitoring system according to claim 1, wherein the output unit outputs an image in which the integrated image has been transformed back to the image before the bird's-eye view was generated.

3. The monitoring system according to claim 1 or 2, wherein the integration unit calculates the actual position indicated by each pixel of the created bird's-eye view based on the position and orientation of the mobile camera, and integrates the detected object to the corresponding position in the bird's-eye view.

4. The monitoring system according to any one of claims 1 to 3, wherein the integration unit generates an image by converting the integrated image back to the image before the bird's-eye view was generated, and the output unit outputs the generated image.

5. The monitoring system according to claim 1, wherein the integration unit identifies the coordinates obtained when the coordinates of the bird's-eye view corresponding to the position of the detected object are transformed back to the image before the bird's-eye view was generated, and integrates the detected object at the position of the identified coordinates in the image.

6. The monitoring system according to any one of claims 1 to 5, wherein the integration unit displays an object that is obscured by an obstacle, as captured by a mobile camera, on the side in front of the obstacle, and generates an image in which the part of the object that is obscured by the obstacle is displayed in a manner different from the other parts of the object.

7. The monitoring system according to any one of claims 1 to 6, wherein the object position calculation unit calculates the position of an object in an image captured by a fixed-point camera, which is a camera set at a fixed position.

8. The monitoring system according to any one of claims 1 to 6, wherein the object position calculation unit calculates the position of an object detected by LiDAR (Light Detection and Ranging) from a fixed position.

9. The monitoring system according to any one of claims 1 to 8, wherein the bird's-eye view creation unit calculates the positions of objects in images captured by a mobile camera, and the integration unit excludes the integration of objects detected by a fixed-point sensor that are located at the same position as objects identified in the bird's-eye view.

10. A monitoring method characterized by calculating the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; creating a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the shooting direction of the mobile camera from above; generating an integrated image, which is an image in which the detected object is integrated into the corresponding position on the bird's-eye view; and outputting the integrated image.

11. A storage medium for storing a monitoring program that causes a computer to perform the following processes: an object position calculation process for calculating the position of an object detected by a fixed-point sensor, which is a sensor installed in a fixed position; a bird's-eye view creation process for creating a bird's-eye view from images captured by a mobile camera, which is a camera mounted on a moving body, showing the shooting direction of the mobile camera from above; an integration process for generating an integrated image, which is an image in which the detected objects are integrated into the corresponding positions on the bird's-eye view; and an output process for outputting the integrated image.

Images