Information processing device, setting support method, and setting support program

Abstract

Provided is an information processing device for providing support for setting a system that monitors the behavior of an object by using a plurality of 3D cameras. This information processing device comprises: a means for reconstructing a 3D space on the basis of respective measurement results from a plurality of 3D cameras in a state where an object of concern existing in a monitoring range of the system is stationary; a means for causing the object of concern to perform a predetermined motion; a means for acquiring point cloud data of the object of concern measured during the motion of the object of concern on the basis of the respective measurement results from the plurality of 3D cameras during the motion of the object of concern; and a means for determining, on the basis of the 3D space and the point cloud data, an area that ceases to be within the monitoring range due to the motion of the object of concern.

Description

Information Processing Apparatus, Setting Support Method and Setting Support Program

[0001] The present invention relates to a system using a plurality of three-dimensional cameras.

[0002] A technique for reconstructing a three-dimensional space based on an image captured by a camera is publicly known. For example, International Publication No. 00 / 07373 (Patent Document 1) discloses an apparatus and a method for displaying a single composite image of a plurality of images captured by several cameras so that the overall state of an area captured by the several cameras can be intuitively understood.

[0003] In addition, Japanese Patent Application Laid-Open No. 2023-102523 (Patent Document 2) discloses a monitoring system and the like that automatically collects position information in a local coordinate system of LiDAR or the like and position information in a local coordinate system of a monitoring camera, and deploys the collected information to a global coordinate system of the entire monitoring space.

[0004] International Publication No. 00 / 07373, Japanese Patent Application Laid-Open No. 2023-102523

[0005] When constructing a system for monitoring the behavior of an object using a plurality of three-dimensional cameras, there is a need to facilitate the construction because the contents to be set become complicated.

[0006] According to an example of the present invention, there is provided an information processing apparatus for supporting setting of a system that monitors the behavior of an object using a plurality of three-dimensional cameras. The information processing apparatus comprises: means for reconstructing a three-dimensional space based on respective measurement results of the plurality of three-dimensional cameras in a state where an object of interest existing in a monitoring range of the system is stationary; means for causing the object of interest to perform a predetermined motion; means for acquiring point cloud data of the object of interest measured while the object of interest is moving, based on respective measurement results of the plurality of three-dimensional cameras obtained while the object of interest is moving; and means for determining a region that is no longer included in the monitoring range due to the motion of the object of interest, based on the three-dimensional space and the point cloud data of the object of interest.

[0007] According to this configuration, it is possible to quantitatively determine how the monitoring range of the system changes when the object of interest moves. This facilitates quantitative risk assessment.

[0008] The determination method may involve determining the occlusion occurring in the field of view of each of multiple 3D cameras based on point cloud data. With this configuration, the area that becomes out of the monitoring range due to the movement of the object of interest can be accurately determined based on the occlusion resulting from the point cloud data of the object of interest measured while the object of interest is moving.

[0009] The means for acquiring point cloud data of an object of interest may determine the point cloud data of the object of interest based on the difference between point cloud data acquired based on the measurement results of each of the multiple 3D cameras when the object of interest is stationary and point cloud data acquired based on the measurement results of each of the multiple 3D cameras when the object of interest is in motion. With this configuration, the point cloud data when the object of interest is in motion can be determined based on the difference in point cloud data relative to the stationary state of the object of interest.

[0010] The information processing device may further include means for displaying areas that are no longer within the monitoring range due to the movement of the object of interest. This configuration allows the user to easily understand the areas that are no longer within the monitoring range due to the movement of the object of interest.

[0011] The object of interest may include a robot. This configuration allows for the evaluation of the impact of a robot that is scheduled to move during the operation.

[0012] Reference markers may be positioned so that they are included in the fields of view of multiple 3D cameras. This configuration can improve recognition accuracy and other aspects.

[0013] According to another example of the present invention, a setup assistance method is provided for assisting in the setup of a system that monitors the behavior of an object using multiple three-dimensional cameras. The setup assistance method includes the steps of: reconstructing a three-dimensional space based on the measurement results of each of the multiple three-dimensional cameras when the object of interest within the system's monitoring range is stationary; causing the object of interest to perform a predetermined action; acquiring point cloud data of the object of interest measured while it is moving, based on the measurement results of each of the multiple three-dimensional cameras while the object of interest is moving; and determining an area that is no longer within the monitoring range due to the movement of the object of interest, based on the three-dimensional space and the point cloud data.

[0014] According to yet another example of the present invention, a setup support program is provided for assisting in the setup of a system that monitors the behavior of an object using multiple three-dimensional cameras. The setup support program causes a computer to perform the following steps: reconstruct a three-dimensional space based on the measurement results of each of the multiple three-dimensional cameras when the object of interest within the system's monitoring range is stationary; cause the object of interest to perform a predetermined action; acquire point cloud data of the object of interest measured while it is moving, based on the measurement results of each of the multiple three-dimensional cameras while the object of interest is moving; and determine an area that is no longer within the monitoring range due to the movement of the object of interest, based on the three-dimensional space and the point cloud data.

[0015] According to the present invention, it is possible to easily construct a system for monitoring the behavior of an object using multiple three-dimensional cameras.

[0016] This is a schematic diagram showing an application example of the safety system according to this embodiment. This is a schematic diagram showing an example configuration of the safety system according to this embodiment. This is a schematic diagram showing an example hardware configuration of the information processing device according to this embodiment. This is a flowchart showing an example of the procedure for configuring the safety system according to this embodiment. This is a schematic diagram showing an example of a reference object including a reference marker used in calibration according to this embodiment. This is a schematic diagram showing an example of a user interface screen for displaying the captured image shown in step S4 of Figure 4. This is a flowchart showing the calibration processing procedure shown in step S5 of Figure 4. This is a schematic diagram showing an example of a user interface screen for calibration shown in step S5 of Figure 4. This is a schematic diagram showing an example of a user interface screen displayed in the monitoring zone setting shown in step S7 of Figure 4. This is a schematic diagram showing an example of a three-dimensional space view included in the user interface screen displayed in the monitoring zone setting shown in step S7 of Figure 4. This is a schematic diagram showing another example of a user interface screen displayed in the monitoring zone setting shown in step S7 of Figure 4. This is a flowchart showing the processing procedure for the coverage rate calculation and display processing shown in step S8 of Figure 4. This is a schematic diagram showing an example of a user interface screen displayed in the coverage rate calculation and display processing shown in step S8 of Figure 4. This is a schematic diagram showing an example of a three-dimensional spatial view included in the user interface screen displayed in the coverage rate calculation and display process shown in step S8 of Figure 4. This is a flowchart showing the processing procedure of the FoV tracking process shown in step S9 of Figure 4. This is a schematic diagram showing an example of a user interface screen showing the navigation function of the FoV tracking process shown in step S9 of Figure 4. This is a flowchart showing the processing procedure of the dynamic blindspot display process shown in step S10 of Figure 4. This is a schematic diagram showing a display example in the dynamic blindspot display process shown in step S10 of Figure 4. This is a diagram for explaining the measurement accuracy confirmation process shown in step S11 of Figure 4. This is a diagram for explaining the measurement accuracy confirmation process shown in step S11 of Figure 4.Figure 4 is a schematic diagram showing an example of a report output in step S12.

[0017] Embodiments of the present invention will be described in detail with reference to the drawings. Note that identical or corresponding parts in the drawings are denoted by the same reference numerals, and their descriptions will not be repeated.

[0018] <A. Application Examples> Below, a safety system will be described as an application example of a system that monitors the behavior of objects using multiple 3D cameras. The behavior of objects includes, for example, changes in position, posture, shape, and state. Specific examples include the movement of people, the movement of products and parts, the operation of machinery and equipment, and the movement of transport vehicles.

[0019] However, the present invention is not limited to "safety systems," but can be applied to any system that monitors the behavior of an object using multiple three-dimensional cameras.

[0020] Figure 1 is a schematic diagram showing an application example of the safety system 1 according to this embodiment. Referring to Figure 1, the safety system 1 is applied to a production system including, for example, one or more collaborative robots. The safety system 1 includes, for example, a plurality of cameras 10-1, 10-2, ... (hereinafter also collectively referred to as "camera 10"). Camera 10 is a three-dimensional camera that images or measures objects present within its field of view (hereinafter also abbreviated as "FoV"). The field of view can be defined as a three-dimensional region. Note that the field of view (FoV) can also be read as the angle of view.

[0021] Camera 10 may be a camera that outputs a 3D image, or it may be a camera that outputs a 2D image (RGB image) and a depth image. Any type of 3D camera is acceptable. For example, it may be a stereo camera system that combines multiple cameras, or a Time of Flight (ToF) system that combines a camera and a light source. Furthermore, the systems may differ among the multiple cameras 10.

[0022] The monitoring range of Safety System 1 is the sum of the FoVs of each of the multiple cameras 10. The monitoring range of Safety System 1 can also be described as the maximum range that Safety System 1 can monitor. The monitoring range of Safety System 1 is determined by factors such as the placement of the multiple cameras 10 and the specifications of each camera 10.

[0023] In the example shown in Figure 1, the monitoring range is the combined FoV 12-1 of camera 10-1 and the FoV 12-2 of camera 10-2. Within this monitoring range, the safety system 1 monitors, for example, the safety distance to the collaborative robot. By integrating the FoVs of multiple cameras 10 to create the monitoring range, blind spots can be reduced, and the flexibility of the equipment layout can be increased.

[0024] <B. Configuration Example> Next, a configuration example of the safety system 1 according to this embodiment will be described.

[0025] Figure 2 is a schematic diagram showing an example configuration of a safety system 1 according to this embodiment. Referring to Figure 2, the safety system 1 includes an information processing device 100 and a plurality of cameras 10-1, 10-2, 10-3, ... The information processing device 100 calculates the position and orientation of objects in the monitoring range and the distance between objects based on the measurement results (e.g., point cloud data) of each camera 10. The information processing device 100 calculates any information necessary for monitoring safety.

[0026] The information processing device 100 may output a safety signal regarding the monitoring range based on the calculated information. For example, the information processing device 100 may output a safety signal when it determines that a preset condition (for example, the distance between the collaborative robot and the person is less than or equal to a predetermined value) has been met. For example, the safety controller may stop or slow down the collaborative robot in response to the safety signal.

[0027] The information processing device 100 may output control commands for controlling devices included in the production system. For example, the control commands may be given to a robot controller that drives a collaborative robot. The robot controller causes the collaborative robot to perform the desired operation according to the control commands.

[0028] <C. Hardware Configuration Example> Next, a hardware configuration example of the information processing device 100 of the safety system 1 according to this embodiment will be described.

[0029] Figure 3 is a schematic diagram showing an example of the hardware configuration of an information processing device 100 according to this embodiment. Referring to Figure 3, the information processing device 100 is an example of a computer and includes one or more processors 102, memory 104, input interface 106, display interface 110, controller interface 114, camera interface 116, network controller 118, and storage 120.

[0030] One or more processors 102 execute computer-readable instructions contained in a program stored in storage 120 to provide the processing and functions described later. When one or more processors 102 execute computer-readable instructions contained in a program, part or all of the program may be loaded into memory 104.

[0031] Storage 120 is an example of a non-transient computer-readable medium. For example, Storage 120 stores an OS (Operating System) 122, a monitoring processing program 124, and a configuration support program 126. OS 122 includes computer-readable instructions for providing the functions necessary for a computer, and computer-readable instructions for generating an environment for executing various programs.

[0032] The monitoring program 124 includes computer-readable instructions for monitoring, as shown in Figure 2.

[0033] The configuration support program 126 includes computer-readable instructions for processing that assists in configuring the safety system 1, as described later.

[0034] The information processing device 100 may store at least one of the monitoring processing program 124 or the setting support program 126.

[0035] The input interface 106 receives input commands from input devices 108 such as keyboards, touch panels, mice, and tablets. The input devices 108 may be part of the configuration of the information processing device 100.

[0036] The display interface 110 outputs video signals to the display 112. The display 112 may be part of the configuration of the information processing device 100.

[0037] The controller interface 114 exchanges data with the safety controller and the robot controller.

[0038] The camera interface 116 exchanges data with multiple cameras 10. The network controller 118 exchanges data with other information processing devices, etc., via the network.

[0039] In this specification, the term "processor" includes not only arithmetic circuits that sequentially execute computer-readable instructions, such as CPUs (Central Processing Units) and GPUs (Graphics Processing Units), but also hardwired circuits. Examples of hardwired circuits include ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays). A "processor" also includes a System on a Chip (SoC) that integrates a core, memory, and peripheral circuits.

[0040] <D. Setting Procedure> Next, an example of a procedure for setting the safety system 1 shown in FIGS. 1 and 2 will be described. As an example, the information processing apparatus 100 can execute processing for supporting the setting of the safety system 1.

[0041] FIG. 4 is a flowchart showing an example of a procedure for setting the safety system 1 according to the present embodiment. The process shown in FIG. 4 may be performed by a user using the information processing apparatus 100 shown in FIGS. 2 and 3, or may be performed using an information processing apparatus different from the information processing apparatus 100 shown in FIGS. 2 and 3. When another information processing apparatus is used, the finally determined setting information may be transferred to the information processing apparatus 100 shown in FIGS. 2 and 3.

[0042] The process executed by the information processing apparatus 100 shown in FIG. 4 may be based on computer-readable instructions included in a setting support program 126.

[0043] In FIG. 4, the process is divided and described in units of steps, but the unit of the process is merely for convenience of description; a plurality of steps may be executed as a single step, or a single step may be further divided into a plurality of steps. A plurality of processes may be executed in parallel, or the execution order between the plurality of processes may be changed.

[0044] Referring to FIG. 4, a user arranges a plurality of cameras 10 (step S1). In addition, the user arranges one or more reference markers for calibration (step S2).

[0045] The information processing apparatus 100 receives an initial setting input by a user (step S3). That is, the user inputs the initial setting to the information processing apparatus 100. The initial setting includes, for example, parameters related to the camera 10 (e.g., number of units, type, identification information, FoV, intrinsic parameter matrix (intrinsic), exposure time, etc.), parameters related to reference markers (e.g., size, calibration data, etc.), selection of a reference camera, and the like.

[0046] The information processing apparatus 100 establishes a connection with the camera 10 according to input initial settings, writes input parameters and the like, and displays an image captured by the camera 10 (step S4). A user may adjust the position of a reference marker while viewing the image captured by the camera 10.

[0047] The information processing apparatus 100 executes calibration in response to a user operation (step S5). A correspondence relationship between the coordinate systems of the camera 10 is determined by the calibration.

[0048] The information processing apparatus 100 gives measurement commands to the plurality of cameras 10, acquires measurement results from each camera 10, generates image data and point cloud data based on the measurement results, and reconstructs a three-dimensional space. Reconstruction of the three-dimensional space includes a process of integrating measurement results acquired from the plurality of cameras 10 to generate three-dimensional data in a unified coordinate system. Reconstruction of the three-dimensional space may include a process of constructing a three-dimensional model from the generated three-dimensional data, and a process of adding surfaces and textures to the generated three-dimensional data.

[0049] Then, the information processing apparatus 100 displays the reconstructed three-dimensional space view (step S6). As described above, the information processing apparatus 100 reconstructs the three-dimensional space based on the measurement results of each of the plurality of cameras 10.

[0050] The information processing apparatus 100 accepts setting of a monitoring zone input by a user in the three-dimensional space (step S7). That is, the user inputs the setting of the monitoring zone to the information processing apparatus 100. The monitoring zone is a range set within the monitoring range of the safety system 1, and may be arbitrarily set by the user according to the purpose. The setting of the monitoring zone includes, for example, selection of a coordinate system used for setting the monitoring zone, the range of the monitoring zone, the level of the monitoring zone (for example, a cooperation zone, a stop zone, etc.), the shape of the monitoring zone, and the like.

[0051] The information processing device 100 performs calculation and display processing of the coverage rate in response to user operation (step S8). The coverage rate represents the proportion of the area occupied by the integrated monitoring range of multiple cameras 10 within a closed three-dimensional area (for example, an area indicating a set monitoring zone). The coverage rate can also be said to represent the proportion of the target three-dimensional area that is included in the FoV of any of the cameras 10. If the coverage rate is 100%, the entire area of ​​the target three-dimensional area is included in the FoV of any of the cameras 10. If the coverage rate is less than 100%, there are areas in the target three-dimensional area that are not included in the FoV of any of the cameras 10 (for example, blind spots).

[0052] In this way, the information processing device 100 calculates the coverage rate, which is the percentage of the area within the monitoring zone that is included in the FoV of any of the multiple cameras 10 in three-dimensional space. Details of the coverage rate and the processing related to the coverage rate will be described later.

[0053] Alternatively, the survival rate may be calculated. The survival rate represents the proportion of the area in the target space that becomes a blind spot.

[0054] The user judges the validity of the set monitoring zone based on the displayed coverage rate. If the coverage rate is lower than the target value, the user may, for example, review the settings of the monitoring zone or review the number, position, and orientation of the cameras 10. At this time, the information processing device 100 performs FoV tracking processing in response to the user operation (step S9). The 3D spatial view is updated sequentially while the FoV tracking processing is being performed. Details of the FoV tracking processing will be described later.

[0055] The information processing device 100 executes dynamic blind spot display processing in response to user operation (step S10). Dynamic blind spot display processing includes processing to determine whether blind spots occur due to movement of an object of interest (e.g., a collaborative robot) within the monitoring range. Details of the dynamic blind spot display processing will be described later.

[0056] The information processing device 100 performs a measurement accuracy verification process in response to user operation (step S11). The measurement accuracy verification process includes a process for calculating and outputting the accuracy of the current calibration. Based on the output calibration accuracy, the user can, for example, determine the validity of the current camera 10 placement. Details of the measurement accuracy verification process will be described later.

[0057] The user may perform some or all of the processes in steps S6 to S11 in any order. Once the position or orientation of the camera 10 is adjusted, calibration (step S5) is performed again, and then some or all of the processes in steps S6 to S10 may be performed in any order.

[0058] After performing the verification steps described above, the user instructs the information processing device 100 to output configuration information and a report. The information processing device 100 outputs a report in response to the user's operation (step S12). Details of the output report will be described later.

[0059] The following will explain some of the processes mentioned above in more detail. <E. Reference Marker (Step S2)> Next, we will explain the reference marker.

[0060] Figure 5 is a schematic diagram showing an example of a reference object including reference markers used in calibration according to this embodiment. Referring to Figure 5, a reference object 20 on which multiple reference markers 22 are arranged may also be used. The reference object 20 is based on a cube. A reference marker 22 is placed on each face of the reference object 20 (a total of 6 faces). Although only three reference markers 22-1 to 22-3 are shown in Figure 5, reference markers 22 are also placed on each of the remaining faces that are not shown.

[0061] Each reference marker 22 contains identification information associated with the surface on which it is placed. Even when multiple cameras 10 image the same reference object 20, the reference markers 22 included in the imaging range can be uniquely identified. Based on the position and identification information of the imaged reference markers 22, and by referring to prior information about the placement of the reference markers 22, the correspondence between the cameras 10 is determined.

[0062] As an example, the AprilTag developed by The APRIL Robotics Laboratory at the University of Michigan can be used as the reference marker 22.

[0063] The reference object 20 has cutouts in parts corresponding to several vertices of a cube. For example, a pair of opposing vertices on the top surface of the reference object 20 have cubic cutouts 24-1 and 24-2. Another pair of opposing vertices on the top surface of the reference object 20 have another pair of cutouts 26-1 and 26-2.

[0064] The notches 24-1, 24-2 and 26-1, 26-2 are geometric features of the reference object 20 and can be features extracted during calibration.

[0065] The calibration used in the safety system 1 according to this embodiment may be based on any algorithm. It is not necessary to use a reference object on which multiple reference markers 22 are arranged; a single reference marker 22 may be placed in any location. Furthermore, the reference object 20 on which multiple reference markers 22 are arranged is not limited to the shape shown in Figure 5, but can be an object of any shape.

[0066] <F. Display of Captured Images (Step S4)> Next, an example of displaying captured images will be described.

[0067] Figure 6 is a schematic diagram showing an example of a user interface screen for displaying the captured image shown in step S4 of Figure 4. The captured image displayed on the user interface screen 200 shown in Figure 6 includes a reference object 20 as the subject. Each of the reference markers 22 provided on the reference object 20 has a guide indicating the recognized position and orientation of each reference marker 22, as well as the recognized identification information of each reference marker 22 superimposed on it.

[0068] When users review the captured images, they can also check the recognized reference markers 22.

[0069] <G. Calibration (Step S5)> Next, an example of the calibration process will be described. Multiple cameras 10 included in the safety system 1 are calibrated.

[0070] During calibration, the information processing device 100 performs alignment (registration or camera registration) between multiple cameras 10 based on the measurement results of the reference markers. For example, if one of the multiple cameras 10 is selected as a reference camera by the user, the information processing device 100 calculates the correspondence between the coordinate system of the reference camera (e.g., the reference camera coordinate system) and the coordinate systems of the other cameras 10 (e.g., the homogeneous transformation matrix, which is an external parameter of the camera 10). By using the correspondence between coordinate systems, the FoV of each of the multiple cameras 10 can be integrated as a monitoring range.

[0071] Furthermore, once the correspondence between the coordinate system of any of the cameras 10 and the world coordinate system (for example, the coordinate system that defines the position and orientation of each piece of equipment) is determined, the monitoring range of the safety system 1 can be handled in the world coordinate system.

[0072] Figure 7 is a flowchart showing the calibration process procedure shown in step S5 of Figure 4. It is assumed that, when performing the calibration, one or more reference markers exist in the FoV of each camera 10.

[0073] Referring to Figure 7, the information processing device 100 issues a measurement command to the camera 10 selected as the reference camera and obtains the measurement result from that camera (step S50). It also issues a measurement command to the camera 10 that is not selected as the reference camera (hereinafter also referred to as the "target camera") and obtains the measurement result from that camera (step S51).

[0074] Thus, in the calibration according to this embodiment, one of the multiple cameras 10 is selected as the reference camera, and the remaining cameras 10 are designated as the target cameras.

[0075] The information processing device 100 generates image data and point cloud data based on the acquired measurement results (step S52). The information processing device 100 recognizes the reference markers based on the acquired measurement results (step S53).

[0076] The information processing device 100 calculates a homogeneous transformation matrix from the reference camera to the reference marker (step S54), and calculates a homogeneous transformation matrix from the target camera to the reference marker (step S55). Using each of these homogeneous transformation matrices, the information processing device 100 calculates a homogeneous transformation matrix from the target camera to the reference camera (step S56). The information processing device 100 calculates the absolute error (step S57). The information processing device 100 displays the calculated absolute error (step S58).

[0077] In step S57, the information processing device 100 calculates the absolute error at the position of a reference marker based on the measurement results of the reference marker within the monitoring range. The absolute error is calculated, for example, by determining multiple point cloud pairs for the same key point between point cloud data based on CAD (Computer Aided Design) information about the reference marker and point cloud data calculated based on the measurement results, and then statistically processing the shape errors of the determined point cloud pairs.

[0078] Figure 8 is a schematic diagram showing an example of a user interface screen for calibration as shown in step S5 of Figure 4. Referring to Figure 8, the user interface screen 200 includes an image 250 captured by the reference camera, point cloud data 252 of the reference marker calculated based on the measurement results of the reference camera, an image 254 captured by the target camera, and point cloud data 256 of the reference marker calculated based on the measurement results of the target camera.

[0079] The user interface screen 200 includes processing results 258, which include point cloud data calculated based on the measurement results of the reference camera and distances based on the recognition results of the reference marker, and processing results 260, which include point cloud data calculated based on the measurement results of the target camera and distances based on the recognition results of the reference marker.

[0080] On the user interface screen 200, once the calibration described above is complete, error information 262 is displayed. The error information 262 includes, for example, absolute error and geometric error.

[0081] Furthermore, all the processes necessary for calibration may be executed with a single user action (for example, by pressing the execute button 264).

[0082] <H. Displaying the 3D spatial view (Step S6) and setting the monitoring zone (Step S7)> Next, an example of the process for displaying the 3D spatial view and setting the monitoring zone will be described.

[0083] Figure 9 is a schematic diagram showing an example of a user interface screen displayed in the setting of a monitoring zone as shown in step S7 of Figure 4. Referring to Figure 9, the user interface screen 200 includes a three-dimensional spatial view 210 reconstructed based on measurement results from multiple cameras 10. Point cloud data is visualized in the three-dimensional spatial view 210 shown in Figure 9. The user interface screen 200 includes an add button 202 and a delete button 204 for monitoring zones. Setting information 206, including parameters related to the cameras 10, is also displayed in the three-dimensional spatial view 210.

[0084] The configuration information 206 may include the selection of a coordinate system for displaying the 3D spatial view 210. For example, the coordinate system may be selected from the camera coordinate system, reference camera coordinate system, world coordinate system, and floor coordinate system (e.g., pull-down 216). The coordinate system may be changed arbitrarily.

[0085] When the add button 202 is selected, the user interface screen 200 accepts the settings for the position and orientation of the monitoring zone to be added. The user interface screen 200 may also accept the settings for the shape and level of the monitoring zone. The user can set a monitoring zone of a desired shape at a desired position and orientation in the 3D spatial view 210. Figure 9 shows the state in which monitoring zone 212 has been set. Multiple monitoring zones 212 may be set.

[0086] When the delete button 204 is selected, the user interface screen 200 accepts the deletion of the previously configured monitoring zone. The user can also change the monitoring zone as needed after it has been configured.

[0087] Figure 10 is a schematic diagram showing an example of a three-dimensional spatial view included in the user interface screen displayed in the monitoring zone setup shown in step S7 of Figure 4.

[0088] Figure 10(A) shows an example of a three-dimensional spatial view based on each axis of the configured coordinate system. In the three-dimensional spatial view 210 shown in Figure 10(A), the user can easily set a monitoring zone parallel to any axis of the coordinate system, for example.

[0089] The information processing device 100 may accept the setting of a monitoring zone while the floor surface recognized based on the measurement results of each of the multiple cameras 10 is displayed. Figure 10(B) shows an example of a three-dimensional spatial view including the recognized floor surface 218. In the three-dimensional spatial view 210 shown in Figure 10(B), the user can, for example, set a monitoring zone parallel to the floor surface 218.

[0090] In any case, the user may adjust the position, orientation, or scale of the monitoring zone after it has been set up.

[0091] Figure 11 is a schematic diagram showing another example of the user interface screen displayed in the monitoring zone setup shown in step S7 of Figure 4. Referring to Figure 11, the floor coordinate system is selected as the coordinate system in the pull-down menu 216 of the user interface screen 200. The floor coordinate system is a coordinate system tangent to the floor surface as the coordinate system in the three-dimensional spatial view 210. When the floor coordinate system is selected, a three-dimensional spatial view 210 that matches the real space is displayed. The three-dimensional spatial view 210 includes a display of the floor surface 218.

[0092] The user interface screen 200 shown in Figure 11 is displayed, allowing the user to easily configure monitoring zones along the floor surface.

[0093] <I. Calculation and Display of Coverage Rate (Step S8)> Next, an example of the process for calculating the coverage rate and an example of the process for displaying the coverage rate will be described. As mentioned above, the coverage rate is the percentage of the area occupied by the monitoring range obtained by integrating the FoV of multiple cameras 10 within a closed three-dimensional area (for example, an area indicating a set monitoring zone).

[0094] Figure 12 is a flowchart showing the processing procedure for calculating and displaying the coverage rate as shown in step S8 of Figure 4. It is assumed that, when performing the calculation and display of the coverage rate, one or more reference markers exist in the FoV of each camera 10.

[0095] Referring to Figure 12, the information processing device 100 issues measurement commands to multiple cameras 10 and acquires measurement results from each camera 10 (step S80).

[0096] The information processing device 100 generates image data and point cloud data based on the acquired measurement results (step S81). The information processing device 100 recognizes the reference marker based on the acquired measurement results (step S82).

[0097] The information processing device 100 calculates the position and orientation of each camera 10, as well as the position and orientation of the reference marker, in the set coordinate system based on the generated image data, point cloud data, and the recognition results of the reference marker (step S83). In this way, the information processing device 100 calculates the position and orientation of each of the multiple cameras 10 based on the measurement results of each of the multiple cameras 10.

[0098] The information processing device 100 updates the 3D spatial view based on the generated image data and point cloud data, the recognition results of the reference markers, the position and orientation of each camera 10, and the calculated position and orientation of the reference markers (step S84).

[0099] Next, the information processing device 100 calculates the coverage rate based on the FoV calculated based on the position and orientation of each of the multiple cameras 10.

[0100] More specifically, the information processing device 100 calculates the FoV of each camera 10 in the set coordinate system based on the position and orientation of each camera 10, as well as the recognition results of the reference marker (step S85). Then, the information processing device 100 calculates the coverage rate of the set monitoring zone based on the calculated FoV of each camera 10 (step S86).

[0101] The information processing device 100 may geometrically calculate the coverage rate from the FoV of each of the multiple cameras 10. For example, the information processing device 100 may geometrically calculate the FoV of each camera 10 based on the position and orientation of each camera 10 and the recognition results for reference markers present in the FoV of each camera 10. For example, the information processing device 100 may determine one or more functions that define the plane or line that forms the boundary of the FoV of each camera 10.

[0102] Furthermore, the information processing device 100 may calculate the coverage rate using the Monte Carlo method. For example, the information processing device 100 may calculate the coverage rate based on the ratio of the number of points included in the FoV of each of the multiple cameras 10 to the number of points included in the monitoring zone. The calculated ratio will represent the coverage rate.

[0103] The FoV of each camera 10 is a three-dimensional area defined in the set coordinate system, and the monitoring zone is a closed three-dimensional area in the set coordinate system. Therefore, the coverage rate can be calculated based on the overlap between the three-dimensional areas.

[0104] If one or more functions are determined that define the plane or line that forms the boundary of the FoV of each camera 10, the intersections of each function with the monitoring zone may be calculated, and the coverage rate may be calculated based on the coordinates of the calculated intersections.

[0105] As will be described later, during the execution of the FoV tracking process, the processes in steps S80 to S86 may be executed repeatedly. By repeatedly executing the processes in steps S80 to S86, the FoV and coverage rate of the camera 10 are updated in real time. Thus, the calculation of position and orientation (step S83), and the calculation of coverage rate (steps S85 and S86) may be executed repeatedly as long as predetermined conditions are met.

[0106] Furthermore, the specified conditions are not limited to the execution of FoV tracking processing; other conditions may also apply.

[0107] Figure 13 is a schematic diagram showing an example of a user interface screen displayed in the calculation and display process of coverage shown in step S8 of Figure 4. Figures 13(A) to 13(D) show several examples with different coverage rates.

[0108] The information processing device 100 may, in displaying the coverage rate, display the range of the monitoring zone and may also display the area included in the FoV of any of the multiple cameras 10 and the other areas in different display modes.

[0109] In the 3D spatial view 220 of the user interface screen 200 shown in Figure 13(A), the configured monitoring zone is not covered at all by the monitoring range which is the combined FoV of the two cameras 10 (coverage rate: 0%).

[0110] In the three-dimensional spatial view 222 of the user interface screen 200 shown in Figure 13(B), the configured monitoring zone is partially covered by the monitoring range that integrates the FoV of the two cameras 10 (coverage rate: 25.277%).

[0111] In the three-dimensional spatial view 224 of the user interface screen 200 shown in Figure 13(C), more than half of the configured monitoring zone is covered by the monitoring range that integrates the FoV of the two cameras 10 (coverage rate: 50.6%).

[0112] In the three-dimensional spatial view 226 of the user interface screen 200 shown in Figure 13(D), the configured monitoring zone is entirely covered by the monitoring range that integrates the FoV of the two cameras 10 (coverage rate: 100%).

[0113] As shown in Figures 13(A) to 13(D), the configured monitoring zone may be displayed in a manner that distinguishes between areas included in the FoV of any of the cameras 10 and areas that are not included. Furthermore, as shown in the three-dimensional spatial views 220, 222, 224, and 226, the FoVs of multiple cameras 10 may be displayed.

[0114] The user interface screens 200 shown in Figures 13(A) to 13(D) may include a numerical display 228 indicating the coverage rate.

[0115] The user can easily understand the coverage rate of the set monitoring zone through the user interface screen 200 shown in Figures 13(A) to 13(D). Furthermore, based on the coverage rate of the set monitoring zone, the user can adjust the position, orientation, shape, and size of the monitoring zone, as well as the position, orientation, and number of cameras 10. This allows for the efficient construction of the safety system 1.

[0116] Furthermore, when conducting a risk assessment, coverage rate can be used as one of the justifications for camera placement.

[0117] Figure 14 is a schematic diagram showing an example of a three-dimensional spatial view included in the user interface screen displayed in the coverage rate calculation and display process shown in step S8 of Figure 4.

[0118] In the three-dimensional spatial view shown in Figure 14(A), the area of ​​the monitoring zone that is included in the monitoring range of the safety system 1 is shown in a similar display manner.

[0119] In the three-dimensional spatial view shown in Figure 14(B), the area of ​​the monitoring zone included in the monitoring range of the safety system 1 is displayed in different display modes depending on the overlap of the FoVs of the cameras 10. For example, an area included in the FoV of only one camera 10 and an area included in the FoVs of any of the multiple cameras 10 may be displayed in different display modes. In this way, the information processing device 100 may display areas of the monitoring zone included in the FoVs of two or more of the multiple cameras 10 in a different display mode than other areas.

[0120] An area that is included in the FoV of any of the multiple cameras 10 can be monitored by the remaining cameras even if occlusion occurs in one camera 10.

[0121] By viewing a three-dimensional spatial view as shown in Figure 14(B), the user can easily confirm the size of the area where monitoring can continue even if occlusion occurs in any of the cameras 10.

[0122] <J. FoV Tracking Process (Step S9)> Next, an example of the FoV tracking process will be described. The FoV tracking process may be performed by the user operating the toggle switch 230 (see Figures 9 and 13, etc.).

[0123] During FoV tracking, if the user changes the position or orientation of any of the cameras 10, the FoV display is updated accordingly. The user can adjust the position or orientation of the cameras 10 while observing how the FoV changes, thereby improving work efficiency.

[0124] Figure 15 is a flowchart showing the processing procedure for the FoV tracking process shown in step S9 of Figure 4. It is assumed that, for the FoV tracking process, one or more reference markers exist in the FoV of each camera 10.

[0125] Referring to Figure 15, the information processing device 100 issues measurement commands to multiple cameras 10 and acquires measurement results from each camera 10 (step S90).

[0126] The information processing device 100 generates image data and point cloud data based on the acquired measurement results (step S91). The information processing device 100 recognizes the reference marker based on the acquired measurement results (step S92).

[0127] The information processing device 100 calculates the position and orientation of each camera 10, as well as the position and orientation of the reference marker, in the set coordinate system based on the generated image data, point cloud data, and the recognition results of the reference marker (step S93). In this way, the information processing device 100 calculates the position and orientation of each of the multiple cameras 10 based on the measurement results of each of the multiple cameras 10.

[0128] The information processing device 100 updates the three-dimensional spatial view based on the generated image data and point cloud data, the recognition results of the reference markers, the position and orientation of each camera 10, and the calculated position and orientation of the reference markers (step S94).

[0129] The processes in steps S90 to S94 may be executed periodically and repeatedly. Furthermore, the processes in steps S90 to S94 may be incorporated into the coverage rate calculation and display process shown in step S8 of Figure 4. That is, the FoV tracking process may be part of the coverage rate calculation and display process, or it may be executed independently of the coverage rate calculation and display process. If the coverage rate calculation and display process and the FoV tracking process are executed independently of each other, the process in the coverage rate calculation and display process may omit processes that are substantially identical to the FoV tracking process.

[0130] The camera 10 can also be guided through the adjustment process using the results of the coverage rate calculation. As the position or orientation of the camera 10 is adjusted, the coverage rate is calculated sequentially. The information processing device 100 can determine the direction of position and orientation that can improve the coverage rate based on the correspondence between the position and orientation of the camera 10 and the calculated coverage rate. For example, the direction in which the peak of the coverage rate exists may be estimated based on the slope of the coverage rate with respect to the position and orientation of the camera 10. The information processing device 100 may graphically display the position and orientation of the camera 10 that can improve the coverage rate.

[0131] Figure 16 is a schematic diagram showing an example of a user interface screen illustrating the navigation function of the FoV tracking process shown in step S9 of Figure 4. Referring to Figure 16, the 3D spatial view 210 of the user interface screen 200 displays a navigation object 240 in the vicinity of one camera 10. The navigation object 240 indicates the direction of movement and rotation, which can improve coverage. The user adjusts the position or orientation of the camera 10 while referring to the navigation object 240.

[0132] Note that the navigation object 240 may be displayed for cameras 10 that are not selected as the reference camera. The navigation object 240 may also be displayed for cameras 10 that are selected as the reference camera, but if the position or orientation of the reference camera changes, calibration will be required again.

[0133] As the user adjusts the position or orientation of camera 10, the 3D spatial view 210 is also updated accordingly, allowing the user to efficiently adjust the placement of camera 10 and the monitoring zone.

[0134] <K. Dynamic Blindspot Display Processing (Step S10)> Next, an example of dynamic blindspot display processing will be described. Dynamic blindspot display processing may be performed by the user operating the dynamic obstacle generation button 208 (see Figure 16).

[0135] As described above, the dynamic blindspot display process includes a process to determine whether blinds will occur due to the movement of an object of interest within the monitoring range.

[0136] Figure 17 is a flowchart showing the processing procedure for the dynamic blind spot display process shown in step S10 of Figure 4. As an example, Figure 17 shows a processing example for confirming that blind spots do not occur due to the operation of a robot.

[0137] Referring to Figure 17, the information processing device 100 issues measurement commands to multiple cameras 10 and acquires measurement results from each camera 10 (step S100).

[0138] The information processing device 100 generates point cloud data and a static map based on the acquired measurement results (step S101). The static map shows the state of objects present in the FoV of each of the multiple cameras 10 and is used to calculate the coverage rate considering obstacles.

[0139] In other words, in step S101, the information processing device 100 reconstructs a three-dimensional space based on the measurement results of each of the multiple cameras 10 while the object of interest is stationary. The reconstructed three-dimensional space shows the state of the monitoring range when a robot, which is an example of an object of interest, is stationary.

[0140] The information processing device 100 issues a control command to the target robot, causing the robot to perform a predetermined operation (step S102). At the same time, the information processing device 100 issues measurement commands to the multiple cameras 10 (step S103).

[0141] The control command may, for example, correspond to the robot's behavior during operation. Instead of the information processing device 100 outputting the control command, the information processing device 100 may provide a trigger signal or the like to a robot controller capable of outputting control commands. Furthermore, the method of causing the object of interest to perform a predetermined action is not limited to using a control command; any method can be used.

[0142] When the robot completes the execution of a predetermined operation, the information processing device 100 acquires measurement results from each camera 10 (step S104), and generates point cloud data based on the acquired measurement results (step S105).

[0143] The information processing device 100 acquires robot motion point cloud data based on the generated point cloud data (step S106). The motion point cloud data represents the point cloud of the robot measured during operation. In this way, the information processing device 100 acquires point cloud data of the object of interest measured during operation based on the measurement results of each of the multiple cameras 10 while the object of interest is in operation.

[0144] One example of a method for acquiring robot motion point cloud data is to use the difference between point cloud data acquired when the robot is stationary and point cloud data acquired when the robot is in motion. For example, the information processing device 100 may determine the robot's motion point cloud data based on the difference between point cloud data acquired when the robot is stationary based on the measurement results of each of the multiple cameras 10 (for example, point cloud data acquired in step S101) and point cloud data acquired when the robot is in motion based on the measurement results of each of the multiple cameras 10 (for example, point cloud data acquired in step S105).

[0145] Alternatively, the information processing device 100 may determine the robot's motion point cloud data based on the temporal changes in point cloud data acquired while the robot is operating. In this case, the robot's motion point cloud data may be determined by the point cloud data that represents the difference between point cloud data acquired at different times while the robot is operating.

[0146] The information processing device 100 calculates blind spots based on the acquired robot motion point cloud data (step S107). That is, the information processing device 100 determines the area that is no longer within the monitoring range due to the robot's movement, based on the three-dimensional space in which the robot is stationary as shown by the static map, and the robot's motion point cloud data.

[0147] As an example of a method for calculating blind spots based on robot motion point cloud data, the information processing device 100 may determine the occlusion occurring in the FoV of each of the multiple cameras 10 based on the robot motion point cloud data. In this case, the information processing device 100 may determine the areas that become invisible from each camera 10 due to the determined occlusion as blind spots.

[0148] The information processing device 100 displays a three-dimensional spatial view including the calculated blind spots (step S108). That is, the information processing device 100 may display areas that are no longer within the monitoring range of the robot (e.g., blind spots).

[0149] Figure 18 is a schematic diagram showing an example of the display in the dynamic blind spot display process shown in step S10 of Figure 4.

[0150] Figure 18(A) shows an example of the robot's motion point cloud data that is acquired. Figure 18(B) shows an example of a three-dimensional spatial view including blind spots. Note that Figure 18(B) shows the three-dimensional regions that constitute blind spots as viewed from two different viewpoints.

[0151] In the dynamic blind spot display process, by actually moving an object (e.g., a robot) that is within the monitoring range, it is possible to confirm whether occlusion may occur because the object obstructs the FoV of camera 10. The position and orientation of camera 10 can be adjusted taking into account the blind spots caused by occlusion.

[0152] Users can also use the results of the dynamic blind spot display process as one of the justifications for camera placement when conducting risk assessments. Furthermore, users can easily confirm that blind spots do not occur even when the robot is in operation.

[0153] Furthermore, in the dynamic blindspot display processing, the reference marker may be positioned so that it is included in the FoV of multiple cameras 10. Including the reference marker in the FoV can improve recognition accuracy.

[0154] <L. Measurement Accuracy Verification Process (Step S11)> Next, an example of the measurement accuracy verification process will be described. The measurement accuracy verification process may include a process to determine the measurement accuracy in the monitoring range of the safety system 1 based on the absolute error for each reference marker placed at different positions within the monitoring range of the safety system 1. The measurement accuracy may include, for example, the absolute error under worst-case conditions.

[0155] For example, the measurement accuracy verification process includes placing reference markers at multiple locations and calculating the absolute error based on each reference marker. Based on the calculated absolute error, the accuracy under worst-case conditions is estimated. That is, the measurement accuracy within the monitoring range of the safety system 1 may be based on the largest value among the calculated absolute errors.

[0156] Figures 19 and 20 are diagrams illustrating the measurement accuracy verification process shown in step S11 of Figure 4.

[0157] Referring to Figure 19, the calculated absolute error differs for both the reference camera and the target camera due to the different positions of the reference markers included in the imaging range. In the example shown in Figure 19, the worst-case value for the reference camera is 33 mm, and the worst-case value for the target camera is also 33 mm. This variation in absolute error is due to factors such as camera aberrations and distance from the camera.

[0158] The information processing device 100 may display candidate locations in the three-dimensional space view where the absolute error is likely to be large within the set monitoring zone. The candidate locations may be determined based on the FoV of the camera 10 and the location of the set monitoring zone, etc. For example, one or more candidate locations may be selected from locations near the FoV boundary of one of the multiple cameras 10, and far from other cameras 10 within the monitoring zone. More specifically, for example, locations that meet the following conditions may be selected.

[0159] - A location near the boundary of the reference camera's FoV (or near the edge of the field of view) and far from the reference camera in the monitoring zone - A location near the boundary of the target camera's FoV (or near the edge of the field of view) and far from the target camera in the monitoring zone The information processing device 100 may perform navigation for measurement accuracy verification processing. For example, as shown in Figure 20, candidate locations where reference markers should be placed are displayed in association with a three-dimensional area representing the monitoring zone 212. That is, the information processing device 100 may display a three-dimensional space including the monitoring range of the safety system 1, and also display one or more candidate locations in the three-dimensional space where reference markers should be placed. The user sequentially places reference markers at the locations in the real space corresponding to the displayed candidate locations. Note that it is not necessary to place reference markers at multiple candidate locations simultaneously.

[0160] The user places a reference marker at a real-space position corresponding to a candidate position, and then operates the information processing device 100 to calculate the absolute error. The calculation of the absolute error may involve repeatedly performing a part of the calibration process shown in Figure 7. Once all reference markers have been placed and the absolute error has been calculated, the user instructs the information processing device 100 to complete the measurement. The information processing device 100 determines the absolute error with the largest value among the absolute errors calculated by the series of measurements as the accuracy under worst conditions.

[0161] The measurement accuracy verification process allows for the sequential calculation of absolute errors for the set monitoring zones, thereby evaluating the measurement accuracy. This enables objective and efficient risk assessment. The accuracy under worst-case conditions may also be included in the report 300 shown in Figure 21.

[0162] Furthermore, during the measurement accuracy verification process, candidate locations for placing the reference marker are presented to the user based on the FoV of the camera 10 and the position of the configured monitoring zone, allowing the user to efficiently perform the measurement accuracy verification process.

[0163] <M. Output of Report (Step S12)> Next, an example of report output will be described. The report output by the information processing device 100 may include, for example, objective data that can be used for risk assessment.

[0164] Figure 21 is a schematic diagram showing an example of a report output in step S12 of Figure 4. Referring to Figure 21, the report 300 includes a three-dimensional spatial view drawn in a predetermined coordinate system. The three-dimensional spatial view shows the FoV of camera 10 and the surveillance zone, etc. The user may arbitrarily select the data to be included in the report 300.

[0165] Report 300 may include data such as the following, for example. However, Report 300 does not need to include all of the data shown below.

[0166] - Coverage rate of each monitoring zone - Survival rate of each monitoring zone - Measurement error of individual cameras - Measurement error when cameras are integrated - Accuracy under worst-case conditions - Case study information of the robot in each monitoring zone. Note that the case study information may include data such as the following. Robot motion data may be prepared separately for each workpiece and approach.

[0167] - Images of voxels containing robot motion data, with the voxels themselves visualized. - Images of the dynamic blind spot area relative to the robot motion data. - The ratio of the dynamic blind spot area relative to the robot motion data relative to the monitoring zone. - Coverage rate of each monitoring zone. - Remaining rate and dynamic remaining rate of each monitoring zone. - Images of the FoV for each camera and the combined FoV of all cameras. - Parameter settings for each camera. By using the report 300 described above, it is possible to provide information that reflects risks, taking into account everything from the layout design of hazards such as robots and sensors such as camera 10 to the actual construction and operation of the equipment. By using the report 300, risk assessment can be made more efficient.

[0168] <N. Modifications> In the above description, an example configuration was shown in which the information processing device 100 performs the main processing. However, multiple information processing devices 100 may work together to perform processing. Alternatively, some or all of the processing may be performed using computing resources on the cloud.

[0169] <O. Addendum> The above-described embodiment includes the following technical concept.

[0170] [Configuration 1] An information processing device (100) for assisting in the setup of a system (1) that monitors the behavior of an object using a plurality of three-dimensional cameras (10), comprising: means (S101) for reconstructing a three-dimensional space based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest within the monitoring range of the system is stationary; means (S102) for causing the object of interest to perform a predetermined operation; means (S104, S105, S106) for acquiring point cloud data of the object of interest measured while the object of interest is in motion, based on the measurement results of each of the plurality of three-dimensional cameras while the object of interest is in motion; and means (S107) for determining an area that is no longer within the monitoring range due to the operation of the object of interest, based on the three-dimensional space and the point cloud data of the object of interest.

[0171] [Configuration 2] The information processing device according to Configuration 1, wherein the means for determining the occlusion occurring in the field of view of each of the plurality of three-dimensional cameras is determined based on the point cloud data.

[0172] [Configuration 3] The information processing apparatus according to Configuration 1, wherein the means for acquiring point cloud data of the object of interest determines the point cloud data of the object of interest based on the difference between point cloud data acquired based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest is stationary and point cloud data acquired based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest is in motion.

[0173] [Configuration 4] The information processing apparatus according to any one of Configurations 1 to 3, further comprising means (S108) for displaying an area that is no longer within the monitoring range due to the movement of the object of interest.

[0174] [Configuration 5] The object of interest is an information processing device according to any one of Configurations 1 to 3, including a robot.

[0175] [Configuration 6] The information processing device according to any one of Configurations 1 to 3, wherein the reference marker is arranged so as to be included in the field of view of the plurality of three-dimensional cameras.

[0176] [Configuration 7] A setting support method for supporting the setting of a system (1) that monitors the behavior of an object using a plurality of three-dimensional cameras (10), comprising: a step (S101) of reconstructing a three-dimensional space based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest within the monitoring range of the system is stationary; a step (S102) of causing the object of interest to perform a predetermined operation; a step (S104, S105, S106) of acquiring point cloud data of the object of interest measured while the object of interest is in operation based on the measurement results of each of the plurality of three-dimensional cameras while the object of interest is in operation; and a step (S107) of determining an area that will no longer be within the monitoring range due to the operation of the object of interest, based on the three-dimensional space and the point cloud data.

[0177] [Configuration 8] A setting support program (126) for assisting in the setup of a system (1) that monitors the behavior of an object using a plurality of three-dimensional cameras (10), wherein the setting support program causes a computer (100) to perform the following steps: reconstruct a three-dimensional space (S101) based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest within the monitoring range of the system is stationary; cause the object of interest to perform a predetermined operation (S102); acquire point cloud data of the object of interest measured while it is operating (S104, S105, S106) based on the measurement results of each of the plurality of three-dimensional cameras while the object of interest is operating; and determine an area that is no longer within the monitoring range due to the operation of the object of interest (S107) based on the three-dimensional space and the point cloud data.

[0178] <P. Advantages> According to the information processing device of this embodiment, it is possible to easily construct a safety system that monitors the behavior of an object using multiple three-dimensional cameras. Furthermore, it is possible to easily perform a risk assessment of the constructed safety system.

[0179] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope of the claims are intended to be included.

[0180] 10 Camera, 20 Reference object, 22 Reference marker, 24, 26 Notch, 100 Information processing device, 102 Processor, 104 Memory, 106 Input interface, 108 Input device, 110 Display interface, 112 Display, 114 Controller interface, 116 Camera interface, 118 Network controller, 120 Storage, 124 Monitoring processing program, 126 Setting support program, 200 User interface screen, 202 Add button, 204 Delete button, 206 Setting information, 208 Dynamic obstacle generation button, 210, 220, 222, 224, 226 3D spatial view, 212 Monitoring zone, 216 Pull-down, 218 Floor surface, 228 Numerical display, 230 Toggle switch, 240 Navigation object, 250, 254 Captured image, 252, 256 Point cloud data, 258, 260; Processing results, 262; Error information, 264; Execute button, 300; Report.

Claims

1. An information processing device for assisting in the setup of a system for monitoring the behavior of an object using multiple three-dimensional cameras, comprising: means for reconstructing a three-dimensional space based on the measurement results of each of the multiple three-dimensional cameras when the object of interest within the monitoring range of the system is stationary; means for causing the object of interest to perform a predetermined action; means for acquiring point cloud data of the object of interest measured while it is moving, based on the measurement results of each of the multiple three-dimensional cameras while the object of interest is moving; and means for determining an area that is no longer within the monitoring range due to the action of the object of interest, based on the three-dimensional space and the point cloud data of the object of interest.

2. The information processing apparatus according to claim 1, wherein the means for determining the occlusion occurring in the field of view of each of the plurality of three-dimensional cameras is determined based on the point cloud data.

3. The information processing apparatus according to claim 1, wherein the means for acquiring point cloud data of the object of interest determines the point cloud data of the object of interest based on the difference between point cloud data acquired based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest is stationary and point cloud data acquired based on the measurement results of each of the plurality of three-dimensional cameras when the object of interest is in motion.

4. The information processing apparatus according to any one of claims 1 to 3, further comprising means for displaying an area that is no longer within the monitoring range due to the movement of the object of interest.

5. The information processing apparatus according to any one of claims 1 to 3, wherein the object of interest includes a robot.

6. The information processing apparatus according to any one of claims 1 to 3, wherein the reference marker is positioned so as to be included in the field of view of the plurality of three-dimensional cameras.

7. A setup support method for assisting the setup of a system for monitoring the behavior of an object using multiple three-dimensional cameras, comprising: reconstructing a three-dimensional space based on the measurement results of each of the multiple three-dimensional cameras when the object of interest within the monitoring range of the system is stationary; causing the object of interest to perform a predetermined action; acquiring point cloud data of the object of interest measured while it is moving, based on the measurement results of each of the multiple three-dimensional cameras while the object of interest is moving; and determining an area that is no longer within the monitoring range due to the movement of the object of interest, based on the three-dimensional space and the point cloud data.

8. A setup support program for assisting in the setup of a system for monitoring the behavior of an object using multiple three-dimensional cameras, the setup support program causing a computer to perform the following steps: reconstruct a three-dimensional space based on the measurement results of each of the multiple three-dimensional cameras when the object of interest within the monitoring range of the system is stationary; cause the object of interest to perform a predetermined action; acquire point cloud data of the object of interest measured while it is moving, based on the measurement results of each of the multiple three-dimensional cameras while the object of interest is moving; and determine an area that is no longer within the monitoring range due to the movement of the object of interest, based on the three-dimensional space and the point cloud data.

Images