Real space monitoring system

Abstract

The present invention comprises: an image data acquisition device 1 that acquires static three-dimensional image data for a real space; a point group data acquisition device 2 that sequentially acquires real-time point group data for the real space; a server device 3 that is connected to the image data acquisition device 1 and the point group data acquisition device 2 in a condition allowing communication; and an information terminal 4 that is connected to the server device 3 in a condition allowing communication, wherein the static three-dimensional data for the real space that has been transmitted from the image data acquisition device 1 and the real-time point group data for the real space that has been transmitted from the point group data acquisition device 2 are combined in the server device 3 and transmitted to the information terminal device 4.

Description

Real - space monitoring system

[0001] The present invention relates to a real - space monitoring system used for monitoring the situation of a real - space by image data and point - cloud data in the real - space.

[0002] In recent years, autonomous driving such as micromobility, mobile robots, and autonomous vehicles has been spreading. For this autonomous driving, in some cases, remote monitoring of visual inspections by humans is required due to legal regulations and the like.

[0003] Generally, in micromobility, mobile robots, autonomous vehicles, etc., various sensors for acquiring various data in the real - space are provided at low positions. Therefore, by providing various sensors on the environmental side such as utility poles and columns in the real - space, it is possible to acquire various data of the entire real - space while covering the blind spots of micromobility, mobile robots, autonomous vehicles, etc.

[0004] And recently, in addition to cameras and millimeter - wave radars, a sensor called LiDAR (light detection and ranging) has attracted attention. This LiDAR is a kind of sensor that uses laser light. Compared with radio waves, the density of the radiation flux is high, and by irradiating a target object while scanning it with a laser light of a short wavelength, in addition to the distance to the target object, the position and shape of the target object can be accurately detected. Therefore, when using LiDAR in a real - space where static objects such as walls and dynamic objects such as automobiles exist, three - dimensional information of the real - space can be acquired as image sensor data consisting of a point cloud of laser light. Moreover, in addition to LiDAR mounted on micromobility, mobile robots, and autonomous vehicles, by using LiDAR arranged in the surrounding environment such as traffic signals on the road side and street corners, these can transmit image sensor data to automobiles to inform the situation and danger of blind spots on the road.

[0005] However, since point cloud data acquired by LiDAR consists only of multiple point clouds of a single color, it has been difficult to grasp the state of static and dynamic objects in real space through human visual inspection. Therefore, a technique has been proposed to combine point cloud data acquired by LiDAR with image data acquired by a camera (see, for example, Non-Patent Document 1).

[0006] K. Kwak et al., "Extrinsic calibration of a single line scanning lidar and a camera," in 2011 IEEE / RSJ International Conference on Intelligent Robots and Systems, 2011, pp. 3283-3289

[0007] However, the images acquired by the camera displayed detailed information about moving objects (such as people's faces and vehicle license plates), which presented a problem in terms of protecting the privacy of these moving objects. Furthermore, at night, the image data acquired by the camera was dark and difficult to see, making it difficult for humans to visually inspect the point cloud data and understand the state of moving and static objects in real space.

[0008] This invention has been made in view of the above-mentioned problems, and aims to provide a real-space monitoring system that allows for easy understanding of the status of dynamic and static objects in point cloud data in real space through human visual inspection, while giving consideration to the privacy of dynamic objects, and thereby enabling easy and reliable monitoring of real space.

[0009] To achieve the above objective, the present invention provides a real-space monitoring system used for monitoring the conditions in real space, comprising: an image data acquisition device for acquiring static image data in real space; a point cloud data acquisition device for sequentially acquiring real-time point cloud data in real space; and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communicative manner, wherein the server device includes an image data storage unit for storing static image data in real space acquired at predetermined timings by the image data acquisition device; a selection unit for selecting point cloud data of a predetermined frame from each frame of real-time point cloud data in real space sequentially acquired by the point cloud data acquisition device; and the image data storage unit. The system is characterized by comprising: a calibrator unit that performs calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the selection unit; an integration unit that generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device with the static image data of a predetermined timing in real space stored in the image data storage unit; and a transmission unit that transmits the integrated image data generated by the integration unit to another device.

[0010] Furthermore, the calibrator unit may use static image data acquired at a time when no moving objects exist in real space for calibration.

[0011] Furthermore, the calibrator unit may use static image data acquired at a time when the real space is bright enough to be visually observed for calibration.

[0012] Furthermore, the selection unit may select point cloud data from each frame of real-time point cloud data acquired sequentially in real space by the point cloud data acquisition device for a predetermined frame in which no dynamic objects exist in real space.

[0013] Furthermore, the calibrator unit may periodically perform calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the point cloud data selection unit.

[0014] Furthermore, when the calibrator unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame, the selection unit may compare the initial frame of the point cloud data selected when the calibration between static image data in real space and real-time point cloud data was first performed by the calibrator unit with each frame of the real-time point cloud data, and select the point cloud data of a predetermined frame in which the position of each point cloud is closest to the point cloud data of the initial frame in real space.

[0015] Furthermore, when the selection unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame using the calibrator unit, it may compare the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q, and select the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space. In this case, the selection unit may select the point cloud data Q of a predetermined frame in which the DM value is smallest in [1] below.

[0016] [Formula 1] P = {p1, p2, p3, ..., pn} Q = {q1, q2, q3, ..., qk} C(q, P): The point in the point cloud P that is closest to q n: The number of points in the point cloud data P k: The number of points in the point cloud data Q

[0017] Furthermore, when the calibrator unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame, the selection unit may compare the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q. When selecting the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space, the selection unit may select the point cloud data Q of a predetermined frame in which the DM value is smallest in [Equation 2] and [Equation 3].

[0018] [Formula 2] [Formula 3] P = {p1, p2, p3, ..., pn} Q = {q1, q2, q3, ..., qk} C(q, P): The point in the point cloud P that is closest to q n: The number of points in the point cloud data P k: The number of points in the point cloud data Q

[0019] Furthermore, the image data acquisition unit may include a camera unit that captures a two-dimensional image of real space, a scanner unit that acquires three-dimensional data of real space, and an image generation unit that generates three-dimensional image data of real space based on the two-dimensional image of real space captured by the camera unit and the three-dimensional data of real space acquired by the scanner unit.

[0020] Furthermore, the integration unit may switch viewpoints in the integrated image data in real space.

[0021] Furthermore, the integration unit may detect dynamic objects in the point cloud data within the integrated image data in real space and continuously or sequentially switch to a viewpoint that tracks the dynamic objects.

[0022] Furthermore, the transmitting unit may preferentially transmit to other devices integrated image data relating to dynamic objects in point cloud data and the surrounding space of said dynamic objects in the integrated image data in real space.

[0023] Furthermore, the integration unit may extract point cloud data limited to a predetermined viewpoint and integrate the point cloud data with the corresponding three-dimensional image data.

[0024] Furthermore, the integration unit may extract point cloud data in which the cosine similarity of the Euclidean distance from the position coordinates of a predetermined viewpoint and / or the direction vector of the predetermined viewpoint falls within a predetermined range.

[0025] Furthermore, the present invention relates to a computer program used in a real-space monitoring system comprising: an image data acquisition device for acquiring static image data in real space; a point cloud data acquisition device for sequentially acquiring real-time point cloud data in real space; and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communication-enabled manner, wherein the server device comprises: an image data storage unit for storing static image data in real space acquired at predetermined timings by the image data acquisition device; a selection unit for selecting point cloud data of a predetermined frame from each frame of real-time point cloud data in real space sequentially acquired by the point cloud data acquisition device; and data stored in the image data storage unit. The system is characterized by comprising: a calibrator unit that performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame in real space selected by the selection unit; an integration unit that generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device with the static image data at a predetermined timing in real space stored in the image data storage unit; and a transmission unit that transmits the integrated image data generated by the integration unit to another device.

[0026] According to the present invention, while respecting the privacy of dynamic objects, the status of dynamic and static objects in point cloud data in real space can be easily grasped by human visual inspection, and consequently, real space can be monitored easily and reliably.

[0027] This figure shows the overall configuration of a real-space monitoring system according to an embodiment of the present invention. This figure shows the configuration of the 3D data acquisition device in Figure 1. This figure shows the configuration of the point cloud data acquisition device in Figure 1. This figure shows the configuration of the server device in Figure 1. This figure shows the configuration of the information terminal device in Figure 1. This figure shows an example of 3D image data according to the first embodiment. This figure shows an example of point cloud data according to the first embodiment. This figure shows a first example of integrated image data according to the first embodiment. This figure shows a second example of integrated image data according to the first embodiment. This figure shows an example of 3D image data according to an embodiment. This figure shows an example of point cloud data according to an embodiment. This figure shows the correlation between DM value, EM value and calibration error. This figure shows the frame-by-frame errors of DM value, EM value and calibration error. This figure shows an example of integrated image data according to the second embodiment. This figure shows the relationship between point cloud data and viewpoint according to the third embodiment. This figure shows an example of integrated image data from a predetermined viewpoint according to the third embodiment.

[0028] <First Embodiment> Next, a first embodiment of the real-space monitoring system according to the present invention (hereinafter referred to as "this system") will be described with reference to Figures 1 to 9. In this embodiment, "real space" refers to a predetermined space where indoor and outdoor monitoring is desired, such as outdoors, such as intersections and roads, or indoors, such as inside buildings and facilities.

[0029] [Overall Configuration] As shown in Figure 1, this system comprises an image data acquisition device 1 that acquires static 3D image data in real space, a point cloud data acquisition device 2 that sequentially acquires real-time point cloud data in real space, a server device 3 connected to the image data acquisition device 1 and the point cloud data acquisition device 2 in a communication-enabled manner, and an information terminal device 4 connected to the server device 3 in a communication-enabled manner. The server device 3 integrates the static 3D image data in real space transmitted from the image data acquisition device 1 and the real-time point cloud data in real space transmitted from the point cloud data acquisition device 2 and transmits it to the information terminal device 4.

[0030] The configurations and functions of the image data acquisition device 1, point cloud data acquisition device 2, server device 3, and information terminal device 4 will be described in detail below.

[0031] [Configuration of Image Data Acquisition Device 1] The image data acquisition device 1 comprises a camera unit 11 that captures a two-dimensional image of real space, a scanner unit 12 that acquires three-dimensional data of real space, an image generation unit 13 that generates static three-dimensional image data of real space based on the two-dimensional image and three-dimensional data of real space, and a transmission unit 14 that transmits the static three-dimensional image data generated by the image generation unit 13 to the server device 3.

[0032] The camera unit 11 is a general-purpose optical machine for imaging a predetermined real space, and it captures a two-dimensional color or monochrome image of the real space. This two-dimensional image may be a static image captured at a predetermined timing, or it may be a static image extracted at a predetermined timing from a continuously captured video.

[0033] The scanner unit 12 is a so-called 3D scanner that acquires three-dimensional data of a predetermined real space, and acquires point cloud data relating to the position and distance of static objects such as walls, corridors, pillars, and furniture that exist in the real space using a laser or the like. This scanner unit 12 may use LiDAR, which will be described later, or other devices.

[0034] The image generation unit 13 generates static three-dimensional image data in real space by combining a two-dimensional image of real space captured by the camera unit 11 and three-dimensional data of real space acquired by the scanner unit 12.

[0035] In this embodiment, as shown in Figure 6, when the image generation unit 13 generates static 3D image data of real space, it generates static 3D image data of real space based on a 2D image of real space captured by the camera unit 11 and 3D data of real space acquired by the scanner unit 12, at a time when no dynamic objects such as vehicles or people are present. As a result, since the static 3D image data of real space does not include dynamic objects such as vehicles or people, the privacy of dynamic objects can be considered.

[0036] Furthermore, when generating static 3D image data of real space, the image generation unit 13 generates static 3D image data of real space based on a 2D image of real space captured by the camera unit 11 and 3D data of real space acquired by the scanner unit 12, at a time when the real space is visible to the human eye in bright conditions (for example, a sunny day during the daytime). As a result, even at dark times such as at night, as will be described later, the status of dynamic and static objects in point cloud data in real space can be easily grasped by human visual inspection.

[0037] [Configuration of Point Cloud Data Acquisition Device 2] The point cloud data acquisition device 2 comprises a sensor unit 21 that acquires point cloud data in real space sequentially in real time, a data processing unit 22 that performs predetermined processing on the point cloud data in real space, and a transmission unit 23 that transmits the point cloud data to the server device 3 sequentially or periodically.

[0038] The sensor unit 21 is a so-called LiDAR (light detection and ranging) sensor. This LiDAR is a type of sensor that uses laser light, and compared to radio waves, it has a higher density of radiant flux. By scanning and irradiating an object with short-wavelength laser light, it acquires image sensor data consisting of point clouds in real space, and accurately detects not only the distance to the object but also the position and shape of the object.

[0039] Examples of LiDAR systems include those that acquire 360-degree omnidirectional point cloud data by rotating multiple laser emitters, and those that acquire image sensor data by directly irradiating with laser light within a predetermined light irradiation angle range. Furthermore, while increasing the number of laser emitters improves the accuracy of the point cloud data, it also increases the cost, so a less expensive LiDAR with fewer emitters may be used.

[0040] The data processing unit 22 performs aggregation and smoothing processing on the point cloud data in real space acquired by the sensor unit 21 at predetermined frame intervals as needed, thereby constructing one frame of point cloud data in real time for each predetermined scan by the sensor unit 21.

[0041] Note that FIG. 7(a) shows an example of point cloud data of a frame in which no dynamic object exists, and FIG. 7(b) shows an example of point cloud data of a frame in which a dynamic object (pedestrian) exists.

[0042] [Configuration of Server Device 3] The server device 3 includes an image data storage unit 31 that stores static three-dimensional image data in the real space, a reception unit 32 that receives real-time point cloud data in the real space, a selection unit 33 that selects point cloud data of a predetermined frame, a calibration unit 34 that performs calibration of the static three-dimensional image data and the point cloud data in the real space, an integration unit 35 that sequentially integrates real-time point cloud data with the static three-dimensional image data in the real space, and a transmission unit 36 that transmits integrated image data in which the static three-dimensional image data and the real-time point cloud data in the real space are integrated to the information terminal device 4.

[0043] The image data storage unit 31 stores static three-dimensional image data at a predetermined timing in the real space transmitted from the image data acquisition device 1. In this embodiment, this three-dimensional image data is at a timing when no dynamic object such as a vehicle or a person exists in the real space as described above, and at least includes static three-dimensional image data acquired at a bright timing when the real space is visible (for example, on a sunny day during the daytime).

[0044] The reception unit 32 receives real-time point cloud data in the real space sequentially or periodically transmitted from the point cloud data acquisition device 2.

[0045] The selection unit 33 selects the point cloud data of a predetermined frame from each frame of the real-time point cloud data in the real space received by the reception unit 32. Specifically, when the calibration between the static three-dimensional image data at a predetermined timing in the real space and the point cloud data of a predetermined frame is executed by the calibrator unit 34, the selection unit 33 selects the point cloud data of a frame in which there are no dynamic objects such as people and vehicles in the real space. According to this, when the calibrator unit 34 executes the calibration between the three-dimensional image data and the point cloud data in the real space as described later, the calibration can be accurately executed under the same or similar conditions where there are no dynamic objects between the three-dimensional image data and the point cloud data in the real space.

[0046] Further, the selection unit 33 compares the first frame of the point cloud data P selected when the calibration between the static three-dimensional image data in the real space and the real-time point cloud data is first executed by the calibrator unit 34 with each frame of the real-time point cloud data Q, and selects the point cloud data Q of a predetermined frame whose position is closest to the point cloud data P of the first frame. According to this, in order to select the real-time point cloud data Q closest to the point cloud data P selected when the calibration is first executed, it is possible to easily select the point cloud data Q of a frame optimal for the calibration in which there are no dynamic objects under the same or similar conditions as the first point cloud data P.

[0047] At this time, the selection unit 33 may select the real-time point cloud data Q in which the DM value becomes the smallest value in the following formula [1].

[0048] [Formula 1] P = {p1, p2, p3,..., pn} Q = {q1, q2, q3,..., qk} C(q, P): The point closest to q in the point cloud P n: The number of points in the point cloud data P k: The number of points in the point cloud data Q Or alternatively, the selection unit 33 may select the real-time point cloud data Q in which the EM value becomes the smallest value in the following formulas [2] and [3].

[0049] [Formula 2] [Formula 3]

[0050] To explain DM and EM in more detail, when the calibrator unit 34 performs the initial calibration of static 3D image data in real space with real-time point cloud data, the frame of point cloud data selected is represented as P = {p1, p2, p3, ..., pn}, where pi (i = 1, 2, 3...) represents an individual point and n is the number of points. In periodic calibration, a predetermined frame of real-time point cloud data is appropriately selected, and this frame is represented as Q = {q1, q2, q3, ..., qk}, where qi (i = 1, 2, 3...) represents an individual point and k is the number of points.

[0051] Furthermore, DM (Distance Metric) is based on the hypothesis that elements shared between two point cloud datasets shorten the distance between nearest neighbors. A smaller DM value indicates more shared elements and more similar frames, which is represented by equation [1] above. Here, C is a function for finding the nearest point in the point cloud.

[0052] On the other hand, Effective Metric (EM) is based on the hypothesis that the more inappropriate points there are in the point cloud, the lower the accuracy of the calibration. A smaller EM value indicates fewer inappropriate points in the frame and is expressed by equation [2] above. Here, D is expressed by equation [3] above.

[0053] Furthermore, during periods when calibration is not being performed (between calibration timings), the selection unit 33 sequentially transmits point cloud data of multiple frames in real space received by the receiving unit 32 to the integration unit 35.

[0054] The calibrator unit 34 performs calibration to align the spatial coordinate axes of static 3D image data of a predetermined timing in real space stored in the image data storage unit 31 and point cloud data of a predetermined frame in real space selected by the selection unit 33.

[0055] Specifically, the calibrator unit 34 first performs calibration in the initial calibration using static 3D image data in real space stored in the image data storage unit 31 at a predetermined timing in real space (a timing when no moving objects exist in real space and the real space is bright enough to be visually observed), and point cloud data of a predetermined frame in real space (a frame in which no moving objects exist) selected by the selection unit 33. At this time, the 3D image data and point cloud data in real space may be from the same timing or from different timings, but it is preferable that both are from a timing when no moving objects exist.

[0056] Furthermore, the calibrator unit 34 periodically performs calibration between static 3D image data at predetermined timings in real space stored in the image data storage unit 31 and point cloud data of predetermined frames in real space selected by the selection unit 33. This is because, when monitoring real space for a long period of time, the coordinates of the 3D image data and point cloud data in real space may shift due to, for example, a shift in the direction or position of the camera unit 11 or scanner unit 12 of the image data acquisition device 1 or the sensor unit 21 of the point cloud data acquisition unit. By periodically performing this calibration between the 3D image data and point cloud data in real space, the integration unit 35 can accurately superimpose and display the point cloud data on the 3D image data in real space, as will be described later. Note that the period in this invention is not necessarily limited to a constant period, and the selected period may be changed midway through the process.

[0057] The integration unit 35 generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device 2 with the static 3D image data at predetermined timings stored in the image data storage unit 31, after the calibration of static 3D image data at predetermined timings and point cloud data for predetermined frames has been performed by the calibrator unit 34. This integrated image data in real space is transmitted sequentially or periodically to the information terminal device 4 by the transmission unit 36.

[0058] For example, Figure 8 shows integrated image data of a dynamic object (pedestrian h) walking in a straight line from the back of the corridor to the front in a real indoor space where the corridor is located at the back and the elevator hall is located at the front. Figure 9 shows integrated image data of a dynamic object (pedestrian h) walking in the elevator hall from the back to the front and then back again.

[0059] Specifically, Figure 8(a) shows integrated image data where no dynamic object exists in real space yet, Figure 8(b) shows integrated image data where a dynamic object (pedestrian h) exists at the far end of the corridor, and Figure 8(c) shows integrated image data where a dynamic object (pedestrian h) exists at the near end of the corridor.

[0060] Furthermore, Figure 9(a) shows integrated image data where a dynamic object (pedestrian h) is present at the back of the elevator hall, Figure 9(b) shows integrated image data where a dynamic object (pedestrian h) is present at the front of the elevator hall, and Figure 9(c) shows integrated image data where a dynamic object (pedestrian h) is again present at the back of the elevator hall.

[0061] According to this method, real-time point cloud data is sequentially superimposed on static 3D image data in a given real space, making it possible to understand the situation of dynamic objects walking in corridors and elevator lobbies within that given space. Moreover, since dynamic objects do not exist in static 3D image data, the status of real-time point cloud data (for example, the movement of point clouds of dynamic objects) can be easily understood on the 3D image in real space. Furthermore, since dynamic objects in point cloud data are simply point clouds and details of people or vehicles cannot be identified, the privacy of dynamic objects can be respected.

[0062] [Configuration of Information Terminal Device 4] The information terminal device 4 is a general-purpose personal computer, tablet terminal, or smartphone, and includes a receiving unit 41 that receives an integrated image which is a combination of static 3D images and point cloud data of the real space transmitted from the server device 3, and a display unit 42 that displays the integrated image of the real space received by the receiving unit 41. This allows the monitor to monitor the real space by viewing the integrated image data of the real space displayed on the display unit 42.

[0063] Next, an embodiment of the first embodiment of this system will be described with reference to Figures 10 to 13.

[0064] Figure 10 visualizes 3D image data of a real-world room. An iPad Pro (iPad is a registered trademark) was used as the image data acquisition device 1, and a LiDAR unit of the point cloud data acquisition device 2 was a Livox Avia. The LiDAR unit operated at 10 fps. 71 frames of data were used for evaluation. These frames consist of 32 frames containing information about dynamic objects (pedestrians) and 39 frames containing only information about static objects. Figure 11 shows the point cloud data frames where pedestrians are present.

[0065] A Point Cloud Library (PCL) was used for calibration. First, downsampling was performed on both the 3D image data and the real-time point cloud data. Next, Fast Point Feature Histograms (FPFH) features were calculated for both sets of data. Then, Sample Consensus Initial Alignment (SAC-IA) was performed for initial rough alignment. Finally, Iterative Closest Point (ICP) was used for precise alignment. The correlation between DM and EM values ​​and the calibration error was investigated. The real-time data was split into two frame sets: point cloud data P for the initial frames and point cloud data Q for the remaining frames. The calibration error is based on the Mean Squared Error (MSE) of the nearest-neighbor point distance between the two point cloud sets. A smaller calibration error indicates higher calibration accuracy.

[0066] Figure 12(a) is a scatter plot of DM and calibration error values, showing a strong correlation with each other, with a correlation coefficient of 0.9934. On the other hand, Figure 12(b) is a scatter plot of EM and calibration error, also showing a strong correlation, with a correlation coefficient of 0.949. Figure 13 shows the values ​​of DM, EM, and calibration error for each frame number, and it can be seen that all three values ​​increase. During the period of these increasing frames, a dynamic object (pedestrian) was walking in the experiment. From the above results, it can be seen that DM is a better metric than EM for frame selection to maximize the accuracy of periodic calibration in the proposed system.

[0067] <Second Embodiment> Next, a second embodiment of the system will be described with reference to Figure 14. In the following, only configurations that differ from the above embodiment will be described, and the same configurations will be omitted from the description and will be denoted by the same reference numerals.

[0068] In this embodiment, the integration unit 35 detects dynamic objects in the point cloud data within the integrated image data in real space and continuously or sequentially switches to a viewpoint that tracks the dynamic objects.

[0069] For example, consider a case where the integration unit 35 generates integrated image data, as shown in Figure 14, in which real-time point cloud data is superimposed onto a static 3D image of a real space (corridor) at a predetermined timing. First, as shown in Figure 14(a), integrated image data is displayed in which the point cloud data of a dynamic object (pedestrian h) is located on the near side of the corridor. Then, as shown in Figure 14(b), integrated image data is successively generated in which the point cloud data of the dynamic object moves towards the far side of the corridor as the dynamic object (pedestrian h) walks further away. At this time, the integration unit 35 detects the dynamic object (pedestrian h) and automatically switches to a viewpoint that tracks the dynamic object (pedestrian h) (a viewpoint behind the dynamic object (pedestrian h)), as shown in Figure 14(c). This makes it possible to monitor the real space while tracking the dynamic object (pedestrian h) in the real space.

[0070] Furthermore, when the transmission unit 36 ​​of the server device 3 transmits integrated image data to the information terminal device 4, it is preferable to prioritize the transmission of integrated image data from the space where the dynamic object exists (the space around the dynamic object). For example, by shortening the transmission period (increasing the transmission frequency) of integrated image data from the space where the dynamic object exists (the space around the dynamic object), and lengthening the transmission period (decreasing the transmission frequency) of integrated image data from other spaces where the dynamic object does not exist, the total amount of integrated image data transmitted can be reduced, and the information terminal device 4 can display the integrated image data sequentially without stress.

[0071] In this embodiment, the viewpoint switching of the integrated image is performed automatically, but it may also be performed manually by a monitor viewing the information terminal device 4.

[0072] <Third Embodiment> Next, a third embodiment of the system will be described with reference to Figures 15 and 16. In the following, only configurations that differ from the above embodiments will be described, and identical configurations will be omitted from the description and given the same reference numerals.

[0073] In this embodiment, when the integration unit 35 generates integrated image data in real space, it extracts point cloud data limited to a predetermined viewpoint and integrates it with the 3D image data corresponding to the point cloud data.

[0074] Specifically, as shown in Figure 15, in three-dimensional real space, the viewpoint is uniquely determined by the position and direction from which it is looking. By automatically or manually determining the viewpoint (position and direction) to be monitored on the screen of the information terminal device 4, the viewpoint is uniquely determined in three-dimensional real space as shown by the arrow in Figure 15, and the integrated image data with the defined viewpoint will look like Figure 16. In this embodiment, the integrated image data is displayed side by side, as shown in Figure 16, with static three-dimensional image data in real space (left figure) and real-time point cloud data (right figure).

[0075] For example, let the viewpoint coordinates be the origin (0,0,0) and the direction of the viewpoint be the vector (X,Y,Z). The direction in which a point (x,y,z) in the point cloud data is visible from the viewpoint is the vector (x,y,z) from the origin. The difference in angle between vector (X,Y,Z) and vector (x,y,z) is generally calculated using cosine similarity. The smaller this cosine similarity, the closer the point is to the viewpoint. Therefore, by extracting points from the point cloud within a predetermined range of cosine similarity, the range of the viewpoint's direction can be defined. Also, the closer a point is to the origin, the closer it is to the viewpoint, and the magnitude of the vector (X,Y,Z), i.e., the Euclidean distance, is X. ２ +Y ２ +Z ２ Since a smaller square root value indicates a point being closer to the viewpoint, the range of distances from the viewpoint can be defined by extracting a point cloud within a predetermined range from a given coordinate position of the viewpoint using Euclidean distance. Note that points that are far from the viewpoint direction (vector), such as those outside the screen, are far from the viewpoint, so cosine similarity takes precedence over Euclidean distance.

[0076] Because the viewpoint's position (coordinate position) and direction (vector) are determined in this way, points that are in the direction (vector) of the viewpoint and are close to the viewpoint's position (coordinate position) are more likely to be important points for the observer, while other points (for example, points on the opposite side of the vector from the viewpoint's coordinate position that are invisible to the observer) are less important. Therefore, it is possible to monitor real space while focusing on important points.

[0077] Although embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiments within the same scope as the present invention, or within an equivalent scope.

[0078] 1…Image data acquisition device 11…Camera unit 12…Scanner unit 13…Image generation unit 14…Transmission unit 2…Point cloud data acquisition device 21…Sensor unit 22…Data processing unit 23…Transmission unit 3…Server device 31…Image data storage unit 32…Receiving unit 33…Selection unit 34…Calibrator unit 35…Integration unit 36…Transmission unit 4…Information terminal device 41…Receiving unit 42…Display unit

Claims

1. A real-space monitoring system used to monitor the conditions in real space, comprising: an image data acquisition device for acquiring static image data in real space; a point cloud data acquisition device for sequentially acquiring real-time point cloud data in real space; and a server device connected to the image data acquisition device and the point cloud data acquisition device in a manner that enables communication, wherein the server device includes: an image data storage unit for storing static image data in real space acquired at predetermined timings by the image data acquisition device; a selection unit for selecting point cloud data of a predetermined frame from each frame of real-time point cloud data acquired sequentially by the point cloud data acquisition device; and a calibrator unit for performing calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the selection unit. A real-space monitoring system characterized by comprising: an integration unit that generates integrated image data by sequentially integrating real-time point cloud data acquired sequentially by a point cloud data acquisition device with static image data of predetermined timings in real space stored in the image data storage unit, after the calibration of static image data of predetermined timings in real space with point cloud data of predetermined frames in real space performed by the calibrator unit; and a transmission unit that transmits the integrated image data generated by the integration unit to another device.

2. The real-space monitoring system according to claim 1, wherein the calibrator unit uses static image data acquired at a time when no moving objects exist in real space for calibration.

3. The real-space monitoring system according to claim 1, wherein the calibrator unit uses static image data acquired at a bright time when the real space is visible for calibration.

4. The real-space monitoring system according to claim 1, wherein the selection unit selects point cloud data of a predetermined frame in which no dynamic objects exist in real space from each frame of real-time point cloud data acquired sequentially in real space by the point cloud data acquisition device.

5. The real-space monitoring system according to claim 1, wherein the calibrator unit periodically performs calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the point cloud data selection unit.

6. The real-space monitoring system according to claim 5, wherein when the calibrator unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame, the selection unit compares the first frame of the point cloud data selected when the calibration between static image data in real space and real-time point cloud data was first performed by the calibrator unit with each frame of the real-time point cloud data, and selects the point cloud data of a predetermined frame in which the position of each point cloud is closest to the point cloud data of the first frame in real space.

7. The real-space monitoring system according to claim 6, wherein the selection unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame using the calibrator unit, compares the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q, and selects the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space, and selects the point cloud data Q of a predetermined frame in which the DM value is smallest in the following [1]. [Formula 1] P = {p1, p2, p3, ..., pn} Q = {q1, q2, q3, ..., qk} C(q, P): The point in the point cloud P that is closest to q n: The number of points in the point cloud data P k: The number of points in the point cloud data Q 8. The real-space monitoring system according to claim 6, wherein the selection unit periodically performs calibration between static image data at a predetermined timing in real space and point cloud data of a predetermined frame using the calibrator unit, and compares the first frame of the point cloud data P selected when the calibration between static image data in real space and real-time point cloud data was first performed with each frame of the real-time point cloud data Q, and selects the point cloud data Q of a predetermined frame in which the position of each point cloud is closest to the point cloud data P of the first frame in real space, and selects the point cloud data Q of a predetermined frame in which the DM value is smallest in [Equation 2] [Equation 3]. [Equation 2] [Formula 3] P = {p1, p2, p3, ..., pn} Q = {q1, q2, q3, ..., qk} C(q, P): The point in the point cloud P that is closest to q n: The number of points in the point cloud data P k: The number of points in the point cloud data Q 9. The real-space monitoring system according to claim 1, wherein the image data acquisition unit comprises a camera unit for capturing a two-dimensional image of real space, a scanner unit for acquiring three-dimensional data of real space, and an image generation unit for generating three-dimensional image data of real space based on the two-dimensional image of real space captured by the camera unit and the three-dimensional data of real space acquired by the scanner unit.

10. The real-space monitoring system according to claim 1, wherein the integration unit switches the viewpoint in integrated image data in real space.

11. The real-space monitoring system according to claim 10, wherein the integration unit detects a dynamic object in the integrated image data in real space and continuously or sequentially switches to a viewpoint that tracks the dynamic object.

12. The real-space monitoring system according to claim 11, wherein the transmitting unit preferentially transmits to other devices integrated image data relating to a dynamic object in point cloud data and the space surrounding the dynamic object in integrated image data in real space.

13. The real-space monitoring system according to claim 1, wherein the integration unit extracts point cloud data limited to a predetermined viewpoint and integrates the point cloud data with corresponding three-dimensional image data.

14. The real-space monitoring system according to claim 13, wherein the integration unit extracts point cloud data in which the cosine similarity of the Euclidean distance from the position coordinates of a predetermined viewpoint and / or the direction vector of the predetermined viewpoint is within a predetermined range.

15. A computer program used in a real-space monitoring system comprising: an image data acquisition device for acquiring static image data in real space; a point cloud data acquisition device for sequentially acquiring real-time point cloud data in real space; and a server device connected to the image data acquisition device and the point cloud data acquisition device in a communication-enabled manner, wherein the server device comprises: an image data storage unit for storing static image data in real space acquired at predetermined timings by the image data acquisition device; a selection unit for selecting point cloud data of a predetermined frame from each frame of real-time point cloud data acquired sequentially by the point cloud data acquisition device; a calibrator unit for performing calibration between static image data of a predetermined timing in real space stored in the image data storage unit and point cloud data of a predetermined frame in real space selected by the selection unit; and, after the calibration between the static image data of a predetermined timing in real space and point cloud data of a predetermined frame has been performed by the calibrator unit, an integration unit for generating integrated image data by sequentially integrating real-time point cloud data acquired sequentially by the point cloud data acquisition device with the static image data of a predetermined timing in real space stored in the image data storage unit. A computer program characterized by functioning as a transmission unit that transmits integrated image data generated by the integration unit to another device.

Images