Environment modeling method, environment modeling device, and program

Abstract

An environment modeling device (1) detects a surface present in a surrounding environment using 3D point cloud data of the surrounding environment obtained through a first measurement by a LiDAR (3), calculates a front position and a normal vector with respect to the detected surface, controls a moving body (2) so as to be moved to the front position, performs a second measurement in which the measurement direction is aligned with the normal vector using the LiDAR (3) and / or radar (4), and creates a 3D model of the surrounding environment using the 3D point cloud data obtained by the first measurement and the second measurement.

Description

Environmental modeling method, environmental modeling apparatus, and program

[0001] The presently disclosed technology relates to an environmental modeling method, an environmental modeling apparatus, and a program.

[0002] In recent years, technologies for creating a three-dimensional (hereinafter referred to as 3D) model of an environment using LiDAR (Light Detection And Ranging) or radar (for example, millimeter-wave radar) have been developed. For example, Patent Document 1 describes a technology for generating a radio wave environment calculation model using LiDAR and a measurement radio device that functions as a radar. The measurement radio device radiates radio waves and receives radio waves reflected by the surface of surrounding structures to identify the direction of arrival of the radio waves. LiDAR irradiates light in the direction of arrival of the radio waves, measures the distance to a point on an object existing in the same direction, and identifies the reflection point of the radio waves radiated by the measurement radio device.

[0003] In Patent Document 1, LiDAR determines the normal of a reflection point assumed to be a surface based on the directions of an incident light wave and a reflected light wave incident on the reflection point on the surface of a structure, and the normal is said to be determined by the fine structure of the reflection point. On the other hand, the radio waves radiated from the measurement radio device have a longer wavelength than the light irradiated by LiDAR, and the reflection point is close to a mirror surface for such radio waves. When an incident radio wave radiated by the measurement radio device is incident on the reflection point, the reflected radio wave is reflected such that the incident direction and the reflection direction form equal angles with respect to the normal of the mirror surface.

[0004] As described above, since the action of the reflection point differs between light waves and radio waves, if the normal of the reflection point is determined using a model created using LiDAR, the direction of a ray simulating the radio wave radiated from the measurement radio device after reflection will differ from that in the actual environment. For this reason, in the technology described in Patent Document 1, the distance to the reflection point is determined by LiDAR, and the reflection characteristics of the reflection point are determined by the radio waves radiated from the measurement radio device.

[0005] Japanese Unexamined Patent Application Publication No. 2024-087719

[0006] The conventional technology described in Patent Document 1 had the problem that there was room for improvement in measuring 3D point clouds of the surrounding environment using LiDAR and radar. For example, although LiDAR can generally measure distance with millimeter (mm) accuracy, the laser light emitted from the LiDAR does not reflect specularly off the surface of the object, so the density of the 3D point cloud decreases depending on the measurement direction. When the 3D point cloud density decreases, it becomes difficult to accurately model the surrounding environment.

[0007] The disclosed technology aims to solve the above-mentioned problems and to provide an environmental modeling method that can accurately model the surrounding environment while suppressing a decrease in 3D point cloud density.

[0008] The environmental modeling method according to the disclosed technology is an environmental modeling method performed by an environmental modeling device that creates a 3D model of the surrounding environment using 3D point cloud data of the surrounding environment obtained by measurement using at least one of LiDAR and radar, which is movable together with a moving object, and comprises the steps of: detecting a surface present in the surrounding environment using 3D point cloud data of the surrounding environment obtained by a first measurement using LiDAR; calculating the position of the front and the normal vector with respect to the detected surface; controlling the moving object to move it to the front position; performing a second measurement using at least one of LiDAR and radar with the measurement direction aligned with the normal vector; and creating a 3D model of the surrounding environment using the 3D point cloud data obtained from the first and second measurements.

[0009] According to the disclosed technology, a surface present in the surrounding environment is detected using 3D point cloud data of the surrounding environment obtained by a first measurement using LiDAR, the position of the front and the normal vector of the detected surface are calculated, a moving object is controlled to move to the front position, and a second measurement is performed using at least one of LiDAR and radar, with the measurement direction aligned with the normal vector, and a 3D model of the surrounding environment is created using the 3D point cloud data obtained from the first and second measurements. As a result, in the environmental modeling method according to the disclosed technology, since both LiDAR and radar can measure the specular reflection component, the decrease in 3D point cloud density is suppressed, and the power of the reflected signal from the 3D point is increased, so measurement with high angular resolution becomes possible. By repeating this operation, the surrounding environment can be modeled with high accuracy.

[0010] Figure 1 is a block diagram showing an example configuration of an environmental modeling device according to Embodiment 1. Figure 2 is a flowchart showing an environmental modeling method according to Embodiment 1. Figure 3 is a schematic diagram showing an overview of the environmental modeling method according to Embodiment 1. Figures 4A and 4B are block diagrams showing the hardware configuration that realizes the functions of the environmental modeling device according to Embodiment 1. Figure 5 is a block diagram showing an example configuration of an environmental modeling device according to Embodiment 2. Figure 6 is a flowchart showing an environmental modeling method according to Embodiment 2. Figure 7 is a schematic diagram showing an overview of the environmental modeling method according to Embodiment 2. Figure 8 is a schematic diagram showing an overview of hole detection in Embodiment 2. Figure 9 is a block diagram showing an example configuration of an environmental modeling device according to Embodiment 3. Figure 10 is a flowchart showing an environmental modeling method according to Embodiment 3.

[0011] Embodiment 1. In the environmental modeling method according to Embodiment 1, a 3D model of the surrounding environment is created using 3D point cloud data of the surrounding environment obtained by measurement using at least one of LiDAR and radar. For example, when the environmental modeling apparatus according to Embodiment 1 acquires 3D point cloud data of the surrounding environment using LiDAR or radar, it reconstructs the shapes of objects present in the surrounding environment by clustering the acquired 3D point cloud and surface reconstruction. Furthermore, the environmental modeling apparatus according to Embodiment 1 identifies the ground, buildings, vehicles, etc., by semantic segmentation and generates a detailed environmental model by applying appropriate textures.

[0012] To accurately model objects in the surrounding environment, a high density of 3D point cloud data is desirable. The higher the 3D point cloud density, the more accurately data can be acquired on the finer details of objects, improving the accuracy of shape reproduction. For example, it becomes easier to identify fine features such as window frames of buildings, unevenness on the sides of vehicles, or branches and leaves of trees. Furthermore, denser 3D point cloud data improves the accuracy of shape reconstruction using clustering or surface reconstruction, resulting in a smoother 3D model with fewer gaps or missing data. In addition, the accuracy of object classification using semantic segmentation is improved, providing more accurate information in areas such as environmental recognition for autonomous driving.

[0013] LiDAR uses laser light to measure the distance to an object with high precision, generally achieving millimeter-level accuracy. However, when the measurement angle is oblique, specular reflection becomes less likely, resulting in decreased performance. This is because when the laser light strikes the surface of an object, if "specular reflection" (reflection at the same angle as the incident angle) does not occur, the receiving sensor cannot detect the reflected light. In particular, on smooth surfaces such as metal or glass, light does not scatter and does not return to the sensor when incident at an oblique angle, making it easy for 3D point cloud data to be lost. When the 3D point cloud density decreases in this way, it becomes impossible to accurately model objects in the surrounding environment.

[0014] In contrast, the environmental modeling method according to Embodiment 1 involves using a LiDAR and radar, which are movable with the moving object, to detect surfaces present in the surrounding environment using 3D point cloud data of the surrounding environment obtained by a first measurement using LiDAR, calculating the front position and normal vector for the detected surface, controlling the moving object to move the LiDAR and radar to the front position, performing a second measurement using at least one of the LiDAR and radar with the measurement direction aligned to the normal vector, and creating a 3D model of the surrounding environment using the 3D point cloud data obtained from the first and second measurements. By executing this method, both LiDAR and radar become capable of measuring specular reflection components, thus suppressing a decrease in 3D point cloud density and increasing the power of the reflected signal from the 3D points. This enables measurements with high angular resolution, and by repeating this operation, objects present in the surrounding environment can be modeled with high accuracy.

[0015] Furthermore, the designations "Part 1" and "Part 2" in this specification are used to identify 3D point cloud measurements and do not necessarily limit the number, order, or content of the components. Also, the numbers used to identify components are used contextually, and a number used in one context does not necessarily indicate the same configuration in another context. Moreover, this does not prevent a component identified by one number from also performing the function of a component identified by another number.

[0016] (Basic Configuration of the Environmental Modeling Device) Figure 1 is a block diagram showing an example configuration of the environmental modeling device 1 according to Embodiment 1, and shows the case where the environmental modeling device 1 is mounted on a mobile device 2. When the environmental modeling device 1 is mounted on a mobile device 2, 3D point cloud data can be acquired while moving, making it possible to scan a wide area of ​​the environment in a short time. For example, by using LiDAR 3 and radar 4 mounted on an autonomous vehicle or drone, the latest 3D maps of urban areas or disaster areas can be generated quickly. In addition, real-time collection and processing of 3D point cloud data becomes possible, which is useful for traffic monitoring, autonomous driving navigation, or infrastructure inspection. Furthermore, integrating it with the position information of the mobile device 2 improves measurement accuracy, thus increasing the accuracy of the 3D model of the environment.

[0017] On the other hand, the environmental modeling device 1 does not necessarily need to be mounted on the mobile body 2. For example, 3D point cloud data acquired by a LiDAR 3 or radar 4 mounted on the mobile body 2 can be transmitted to the environmental modeling device 1 on the cloud or remotely to generate a 3D model of the environment. Furthermore, by sharing 3D point cloud data acquired by another mobile body 2, it is possible to create a high-quality 3D model. Moreover, by having multiple mobile bodies 2 cooperate to acquire data and integrate it, more comprehensive and accurate environmental modeling becomes possible.

[0018] The environmental modeling device 1 creates a 3D model of the surrounding environment using 3D point cloud data of the surrounding environment obtained by measurements using at least one of LiDAR 3 and radar 4. The 3D model of the surrounding environment created by the environmental modeling device 1 is sent to various output destinations depending on the application. For example, if the 3D model is used as map data for vehicle navigation or obstacle avoidance, the 3D model is output to an autonomous driving system. The 3D model of the surrounding environment is also used in games or simulations on AR / VR platforms. In the fields of architecture and civil engineering, the 3D model of the surrounding environment is imported into BIM (Building Information Modeling) software and used for analysis for urban planning or architectural design.

[0019] (Mobile Entity) Mobile entity 2 is a mobile object equipped with the environmental modeling device 1, LiDAR 3, and radar 4. For example, various things can be considered as mobile entity 2. Cars or drones are typical mobile entities. By mounting the environmental modeling device 1, LiDAR 3, and radar 4 on a car or drone, urban environments or road conditions can be scanned in real time. By mounting the environmental modeling device 1, LiDAR 3, and radar 4 on a robot or automated guided vehicle (AGV), automated driving in warehouses or environmental recognition within facilities becomes possible. It can also be mounted on railways or ships and used for applications such as inspecting railway infrastructure or marine exploration. Mobile entity 2 does not necessarily have to be a machine; it is also conceivable that a person could carry the environmental modeling device 1, LiDAR 3, and radar 4. For example, if a worker wears a backpack-type 3D scanner and takes measurements while walking, a detailed environmental model of a construction site or the inside of a cave can be created. In this way, mobile entity 2 is diverse, and even when a person becomes mobile entity 2, it remains an effective means of environmental modeling. Figure 1 shows the case where the mobile unit 2 is a vehicle including an AGV.

[0020] (LiDAR) LiDAR3 uses laser light to measure the distance to an object and acquires 3D point cloud data of the environment. Its basic principle is to calculate 3D point data including distance information to the object by measuring the time (round trip time) it takes for the laser light irradiated onto the object to reflect off the object and return. Specifically, LiDAR3 irradiates the surrounding environment with laser pulses and detects the reflected light that returns after the laser light is reflected off the surfaces of objects in the surrounding environment using an internal light receiving sensor. Then, LiDAR3 measures the time from when the laser light is irradiated until it returns and calculates the distance using the speed of light.

[0021] Furthermore, the advantages of LiDAR3 lie in its high-precision distance measurement and environmental recognition capabilities. Because it uses laser light for measurement, it can acquire the distance to an object with high precision down to the millimeter, enabling the generation of detailed 3D point cloud data. For example, 3D mapping is possible, and the surrounding environment can be visualized using the 3D point cloud data. LiDAR3 also has the advantage of high resolution. With LiDAR3, fine shapes can be recognized, and the contours of objects can be grasped in detail. In addition, LiDAR3 can perform high-precision measurements in a range of approximately 5m to 200m.

[0022] On the other hand, LiDAR3 has limitations in measuring certain objects. For example, with objects that have high laser light transmittance, such as glass, the laser light passes through the object, resulting in insufficient reflected waves and making measurement difficult. Objects with low light reflectance are also difficult to measure. For instance, black objects with a matte finish have low light reflectance. Furthermore, LiDAR3 is susceptible to weather conditions such as rain, fog, or snow, which can scatter the laser light and reduce measurement accuracy. Additionally, when the laser light is shone obliquely onto the surface of an object, the density of the 3D point cloud decreases. In other words, LiDAR3 does not utilize specular reflection, where the component of obliquely incident laser light reflected in the same direction is small, thus reducing the reflected power of the laser and decreasing accuracy. In LiDAR3, acquiring 3D point cloud data becomes difficult when measuring an object from an oblique direction, or when the object has high or low laser light transmittance.

[0023] (Radar) Radar 4 measures the distance, relative velocity, or direction to an object using electromagnetic waves (e.g., millimeter waves). The operating principle is that Radar 4 emits electromagnetic waves, receives the signal reflected by the object, and calculates the distance to the object by measuring the time difference (round-trip time). Furthermore, the object's moving velocity can be detected by utilizing the Doppler effect. In addition, by using phased array technology, it is possible to control multiple beams simultaneously and accurately determine the direction of the object.

[0024] The advantages of radar 4 are as follows. For example, if radar 4 is a millimeter-wave radar, it is less affected by fog, rain, snow, or smoke, so stable measurements are possible even in adverse weather conditions. Furthermore, detection is possible even at night. In addition, because millimeter waves have a relatively short wavelength, radar 4 can detect objects tens to hundreds of meters away. Moreover, unlike visible light or infrared light, radar 4 has the characteristic of not easily penetrating glass or water. For this reason, radar 4 can receive and detect radio waves reflected by glass or water. The environmental modeling device 1 measures the surrounding environment by taking advantage of the advantages of LiDAR 3 and radar 4 described above.

[0025] (Mobile Body Drive System) The mobile body drive system 5 is a device that moves the mobile body 2 according to information acquired from the movement control unit 15, and is configured, for example, with a motor control unit, a communication interface, and a navigation system. It analyzes the front position and normal vector information acquired from the movement control unit 15, determines the path to avoid obstacles or to the destination, and performs appropriate acceleration, deceleration, or turning of the mobile body 2.

[0026] (Sensor Driving Device) The sensor driving device 6 is a device that adjusts the attitude (direction and angle) of the LiDAR 3 and radar 4 according to the information acquired from the movement control unit 15. By adjusting the attitude of the LiDAR 3 and radar 4 with the sensor driving device 6, the LiDAR 3 and radar 4 are directed in the optimal direction, enabling the acquisition of accurate environmental data. The main components of the sensor driving device 6 include a motor control unit, gimbal, rotation mechanism, and communication interface. For example, the LiDAR 3 on a vehicle has its angle corrected by the sensor driving device 6 according to the vehicle's tilt before taking measurements. Also, the radar 4's beam axis is dynamically adjusted by the sensor driving device 6, enabling efficient measurements.

[0027] As shown in Figure 1, the environmental modeling device 1 includes a measurement control unit 11, a data acquisition unit 12, a surface detection unit 13, a normal vector calculation unit 14, a movement control unit 15, a storage unit 16, and a model creation unit 17. For example, the environmental modeling device 1 is implemented by a computer. The memory of this computer stores programs that constitute information processing applications for realizing each of the functions of the measurement control unit 11, the data acquisition unit 12, the surface detection unit 13, the normal vector calculation unit 14, the movement control unit 15, the storage unit 16, and the model creation unit 17. The processor of the computer executes the information processing applications read from the memory, thereby realizing each of the functions of the measurement control unit 11, the data acquisition unit 12, the surface detection unit 13, the normal vector calculation unit 14, the movement control unit 15, the storage unit 16, and the model creation unit 17.

[0028] (Measurement Control Unit) The measurement control unit 11 controls whether to perform measurements using LiDAR 3 or radar 4. For example, the measurement control unit 11 performs a first measurement using LiDAR 3, and when the moving object 2 moves to a position facing the surface, it performs a second measurement using at least one of LiDAR 3 and radar 4, with the measurement direction aligned with the normal vector of the surface. Specifically, when the first measurement of the surrounding environment of the moving object 2 is performed by LiDAR 3, the moving object 2 moves to a position facing the surface calculated using the 3D point cloud data obtained in the first measurement. After this, the attitude of at least one of LiDAR 3 and radar 4 is controlled so that its measurement direction is aligned with the normal vector of the surface. When this is notified by the movement control unit 15, the measurement control unit 11 starts a second measurement using at least one of LiDAR 3 and radar 4.

[0029] The "measurement direction" refers to the direction along the optical axis of LiDAR 3 and the beam axis of radar 4. Furthermore, "aligning the measurement direction with the surface normal vector" means making the measurement direction and the direction of the normal vector coincide within a predetermined tolerance range. When the LiDAR and radar 4 are installed in close proximity, and the measurement directions of LiDAR 3 and radar 4 are approximately coaxial, it is possible to align the measurement directions of both LiDAR 3 and radar 4 with the normal vector.

[0030] By aligning the measurement direction of at least one of LiDAR3 and radar4 with the normal vector of the surface, the laser light from LiDAR3 and the radio waves from radar4 become reflected signals with respect to the surface as specular reflection components. Generally, the specular reflection component can be the strongest reflected signal, resulting in a high SNR (Signal-to-Noise Ratio) of the distance measurement signal and obtaining accurate measurement results. Furthermore, the angular resolution is highest when measured from the front, so the 3D point cloud data obtained here is denser and more accurate than the 3D point cloud measured from an arbitrary location in the first measurement.

[0031] (Data Acquisition Unit) The data acquisition unit 12 acquires 3D point cloud data obtained from a first measurement by LiDAR 3, and further acquires 3D point cloud data obtained from a second measurement by at least one of LiDAR 3 and radar 4. The data acquired from LiDAR 3 and radar 4 by the data acquisition unit 12 is "point cloud" data consisting of a large number of 3D coordinate points (X, Y, Z). In addition, since 3D point cloud data may contain noise, the data acquisition unit 12 may perform preprocessing to remove the noise. As preprocessing, for example, statistical outlier removal is performed to remove measurement errors or abnormal values. The data acquisition unit 12 may also divide the 3D point cloud data into cubes (voxels) according to their number. The 3D point cloud data acquired by the data acquisition unit 12 is stored in the storage unit 16.

[0032] (Surface Detection Unit) The surface detection unit 13 detects surfaces present in the surrounding environment using 3D point cloud data of the surrounding environment obtained from the first measurement by LiDAR3. For example, the surface detection unit 13 detects surfaces using the Ball-Pivoting algorithm (hereinafter referred to as "BPA") or the Poison Reconstruction algorithm (hereinafter referred to as "PRA"). BPA is a sequential mesh generation method that forms a triangular mesh using adjacent points of 3D points. The basic processing flow is as follows: (1) Select three initial points to create a triangle, (2) Roll a virtual sphere on the 3D point cloud to expand the triangle, and (3) Construct a mesh by adding new points. BPA detects the completed mesh as a plane by repeatedly performing processes (1) to (3) on the entire 3D point cloud.

[0033] PRA is an algorithm that reconstructs a smooth surface from 3D point cloud data. PRA creates a continuous surface (mesh) based on the 3D point cloud. The basic processing flow is as follows: (1) acquire 3D point cloud data, (2) output the acquired 3D point cloud data to the normal vector calculation unit 14 to calculate the normal vector, and (3) construct the mesh by Poisson reconstruction using the calculated normal vector information. PRA detects the completed mesh as a plane by repeatedly performing processes (1) to (3) on the entire 3D point cloud. In Poisson reconstruction, a smooth mesh is reconstructed by solving the Poisson equation using the 3D point cloud as the gradient vector of a scalar field. The normal vector of each triangle in the mesh is acquired, and triangles with matching normal vectors are extracted as a plane.

[0034] (Normal Vector Calculation Unit) The normal vector calculation unit 14 calculates the front position and normal vector for the face detected by the face detection unit 13. For example, the normal vector calculation unit 14 identifies the center position of each of the multiple faces detected by the face detection unit 13 and determines the position of the measurement point relative to these faces. In the case of a triangular mesh face, the centroid of its vertices can be approximated as the center position of the face. Next, if a measurement point S is given, the normal vector N of the face is used to determine the direction in which the measurement point is relative to the face. The distance between the measurement point and the face is obtained by calculating the dot product d of the vector component V from point S to the center C of the face with the normal vector. The sign of d allows us to determine whether the measurement point S is in the direction of the normal vector or the opposite direction. If d > 0, it is in the direction of the normal vector, and if d < 0, it is located on the back side. Then, the normal vector calculation unit 14 calculates the front position P of the face as a point moved by an appropriate distance in the direction of the normal vector from the measurement point S according to the following formula (1). In the following equation (1), k is an arbitrary scalar quantity that is adjusted according to the appropriate distance. P = C + kN (1)

[0035] To determine the normal vector of a surface, a vector calculation is performed using the coordinate information of the 3D points that make up the surface. For example, the normal vector calculation unit 14 calculates two edge vectors from the vertex coordinates of the surface and calculates the normal vector by taking their cross product. Then, the normal vector calculation unit 14 normalizes the calculated normal vector to a unit vector to calculate a normal vector that represents only the direction. Using the 3D point cloud data obtained in the first measurement, the normal vector calculation unit 14 calculates the front position and the normal vector, which is the orientation, of various surfaces present in the surrounding environment. The environment modeling device 1 performs a second measurement for each of these surfaces.

[0036] (Movement Control Unit) The movement control unit 15 moves the mobile body 2 based on the information of the front position and normal vector of the surface calculated by the normal vector calculation unit 14, and adjusts the attitude of the LiDAR 3 and radar 4. For example, if the mobile body 2 is a vehicle (AGV) or a drone, the movement control unit 15 calculates the optimal path from the current position P0 of the mobile body to the target position P. For example, it checks for the presence of obstacles and determines an avoidance path if necessary. The movement control unit 15 instructs the mobile body drive unit 5 to control the motor or steering according to the determined path and moves the mobile body 2 to the target position P. Near the target position, the mobile body 2 is decelerated and finally its position is adjusted within a predetermined error range and stopped.

[0037] When the mobile body 2 moves to the front position P, the movement control unit 15 instructs the sensor drive unit 6 to align the measurement direction of at least one of the LiDAR 3 and radar 4 with the normal vector of that surface. For example, the movement control unit 15 measures the current orientation (measurement direction) of the LiDAR 3 and radar 4 using an IMU (Inertial Measurement Unit) or encoder and calculates a vector V indicating the current measurement direction. The movement control unit 15 then calculates the angle θ of vector V with respect to the normal vector N, and further calculates the rotation axis R for aligning these directions from the cross product of V and N. The movement control unit 15 outputs the angle θ and rotation axis R as control information for attitude adjustment to the sensor drive unit 6. Based on the control information from the movement control unit 15, the sensor drive unit 6 rotates using a gimbal or motor so that the measurement direction matches the normal vector N. At this time, PID control may be used to perform smooth and highly accurate attitude adjustment.

[0038] (When the moving object is a person) When the moving object 2 is a person, the environment modeling device 1, LiDAR 3, and radar 4 are realized as a device that can move with the person. This device does not have the moving object drive device 5 and sensor drive device 6 shown in Figure 1; the person moves, and the person controls the attitude of LiDAR 3 and radar 4. For example, if the device has a display monitor, the movement control unit 15 calculates the optimal movement route to the target position P and displays an arrow or guideline on the display monitor. If AR technology is used, the direction or distance to the target position P may be overlaid within the field of view to provide intuitive movement guidance. When the moving object 2 moves to the front position P, the movement control unit 15 displays the normal vector N on the device's monitor. For example, a guideline indicating the direction of the normal vector N may be displayed on the tablet screen to allow manual adjustment.

[0039] (Storage Unit) The storage unit 16 is a storage unit that stores the 3D point cloud data acquired by the data acquisition unit 12. For example, in the storage unit 16, newly obtained 3D point cloud data in a common surrounding environment is superimposed on the 3D point cloud data obtained in the first measurement. That is, the data contains the 3D point cloud data obtained in both measurements. By repeating the second measurement for all surfaces present in the surrounding environment, high-density and high-precision 3D point cloud data can be obtained for all surfaces. If the 3D point cloud data is high-density and high-precision, the surfaces (model mesh) detected from the 3D point cloud data will also be highly accurate.

[0040] Figure 1 shows a storage unit 16 built into the environmental modeling device 1, but it is not limited to this. For example, the storage unit 16 may be an external storage device provided separately from the environmental modeling device 1. In this case, the data acquisition unit 12 and the model creation unit 17 access the storage unit 16, which is an external storage device, using a communication device (not shown in Figure 1) to exchange 3D point cloud data.

[0041] (Model Creation Unit) The model creation unit 17 creates a 3D model of the surrounding environment using the 3D point cloud data obtained from the first and second measurements. For example, the model creation unit 17 reduces measurement errors by performing noise reduction and outlier correction on the 3D point cloud data. Then, the model creation unit 17 performs semantic segmentation on multiple surfaces detected by the surface detection unit 13 and identifies objects such as the ground, buildings, and trees, thereby constructing a highly accurate 3D model of the environment. In this way, the environment modeling device 1 easily models the environment using the 3D point cloud data obtained from the first measurement, and then uses the results to perform a highly accurate second measurement, thereby suppressing a decrease in 3D point cloud density and enabling accurate modeling of objects present in the surrounding environment.

[0042] (Environmental Modeling Method) Next, the environmental modeling method according to Embodiment 1 will be described. Figure 2 is a flowchart showing the environmental modeling method according to Embodiment 1, and shows a series of operations by the environmental modeling device 1 shown in Figure 1. The measurement control unit 11 causes the LiDAR 3 to perform a first measurement of the surrounding environment (step ST1). The 3D point cloud data obtained from the first measurement is acquired by the data acquisition unit 12 and output to the surface detection unit 13 and the storage unit 16.

[0043] The surface detection unit 13 uses the 3D point cloud data obtained from the first measurement to detect the surfaces of objects in the surrounding environment (step ST2). The surface information detected by the surface detection unit 13 is output to the normal vector calculation unit 14. The normal vector calculation unit 14 calculates the normal vector of the surface based on the surface information (step ST3). When detecting a surface using a candidate normal vector, the surface detection unit 13 and the normal vector calculation unit 14 work together to calculate the normal vector and detect the surface based on the calculation result. In other words, the processes of step ST2 and step ST3 may be executed as a single process.

[0044] The movement control unit 15 determines whether or not the second measurement has been performed from the front positions of all surfaces by at least one of the LiDAR 3 and the radar 4 (step ST4). If it is determined that there is a surface on which the second measurement from the front has not been performed (step ST4; NO), the movement control unit 15 controls the mobile body driving device 5 to move the mobile body 2 to the front of one of the remaining surfaces (step ST5). After that, the movement control unit 15 controls the sensor driving device 6 to adjust the measurement direction of at least one of the LiDAR 3 and the radar 4 to match the normal vector. When this adjustment is completed, the movement control unit 15 notifies the measurement control unit 11 that the preparation for the second measurement is completed.

[0045] When receiving the above notification from the movement control unit 15, the measurement control unit 11 causes the second measurement, which is re-measurement of the point cloud of the surface using at least one of the LiDAR 3 and the radar 4, to be performed (step ST6). The 3D point cloud data obtained by the second measurement is acquired by the data acquisition unit 12 and output to the storage unit 16. Accordingly, the storage unit 16 stores, as data obtained by superimposing the 3D point cloud data newly obtained in the second measurement on the 3D point cloud data obtained in the first measurement (step ST7). After that, the process returns to step ST4, and the processes from step ST5 to step ST7 are executed on the remaining surfaces.

[0046] On the other hand, when the second measurement has been performed on all surfaces (step ST4; YES), the model creation unit 17 creates a 3D model of the surrounding environment using the 3D point cloud data stored in the storage unit 16 (step ST8). When the environmental modeling apparatus 1 executes the processes from step ST1 to step ST8, both the LiDAR 3 and the radar 4 can measure specular reflection components, the reduction in 3D point cloud density is suppressed, and the power of the reflection signal from the 3D points is increased, thereby enabling measurement with high angular resolution. By repeating this operation, the surrounding environment can be modeled with high accuracy.

[0047] Next, an overview of the environment modeling method according to the first embodiment will be described. FIG. 3 is a schematic diagram showing an overview of the environment modeling method according to the first embodiment, and illustrates a case where a measurement object 30 is measured. In FIG. 3, it is assumed that the measurement object 30 is a cube, and the length of one side thereof is d. For example, as shown on the left side of FIG. 3, when measurement (first measurement) is performed using LiDAR 3 from a position in front of a surface of the measurement object 30, the 3D points 31 on this surface have a high point cloud density. On the other hand, a surface adjacent to this surface is measured from an oblique direction by LiDAR 3, so the point cloud density of the 3D points 31 decreases.

[0048] In contrast, as shown on the right side of FIG. 3, the environment modeling apparatus 1 moves the moving body 2 to the front position of the surface measured from the oblique direction in the first measurement, and performs the second measurement by aligning the measurement direction with the normal vector of the surface. As a result, high-density 3D point cloud data can be obtained from the surface subjected to the second measurement, similar to the surface subjected to the first measurement from the front. Furthermore, in the second measurement, when a surface that is measured from an oblique direction is detected, the second measurement is regarded as the first measurement, and the second measurement is performed on the newly detected surface. By repeating these processes, it is possible to acquire high-density and high-precision 3D point cloud data from the surrounding environment.

[0049] Next, a hardware configuration that implements the functions of the environment modeling apparatus 1 will be described. The functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17 provided in the environment modeling apparatus 1 are implemented by a processing circuit. That is, the environment modeling apparatus 1 includes a processing circuit for executing the processing from step ST1 to step ST8 shown in FIG. 2. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in a memory.

[0050] Figure 4A is a block diagram showing the hardware configuration for realizing the functions of the environmental modeling device 1. Figure 4B is a block diagram showing the hardware configuration for executing the software that realizes the functions of the environmental modeling device 1. In Figures 4A and 4B, the data acquisition unit 12 shown in Figure 1 acquires measurement data from the LiDAR 3 and radar 4 via the input interface 100 and stores the acquired data in the storage unit 16. The model creation unit 17 outputs the created 3D model of the surrounding environment to the outside via the output interface 101.

[0051] If the processing circuit is a dedicated hardware processing circuit 102 as shown in Figure 4A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17 of the environmental modeling device 1 may be implemented by separate processing circuits, or these functions may be implemented together in a single processing circuit.

[0052] When the processing circuit is the processor 103 shown in Figure 4B, the functions of the environmental modeling device 1, including the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17, are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.

[0053] The processor 103 reads and executes a program stored in the memory 104, thereby realizing the functions of the environmental modeling device 1, namely the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17. For example, the environmental modeling device 1 includes a memory 104 for storing a program that, when executed by the processor 103, will result in the execution of steps ST1 to ST8 shown in Figure 2. These programs cause the computer to execute the procedures or methods of processing performed by the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17. The memory 104 may also be a computer-readable storage medium that stores a program for causing the computer to function as the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17.

[0054] The memory 104 may include, for example, non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM) (registered trademark), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, and DVDs. The memory 104 may also function as a storage unit 16.

[0055] Furthermore, some of the functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15, storage unit 16, and model creation unit 17 of the environmental modeling device 1 may be implemented by dedicated hardware, while other parts may be implemented by software or firmware. For example, the functions of the data acquisition unit 12 and the normal vector calculation unit 14 may be implemented by a processing circuit 102, which is dedicated hardware, while the functions of the measurement control unit 11, surface detection unit 13, movement control unit 15, storage unit 16, and model creation unit 17 may be implemented by a processor 103 reading and executing a program stored in memory 104. In this way, the processing circuit can implement the above functions by hardware, software, firmware, or a combination thereof.

[0056] As described above, the environmental modeling method according to Embodiment 1 is an environmental modeling method performed by an environmental modeling device 1 that creates a 3D model of the surrounding environment using 3D point cloud data of the surrounding environment obtained by measurement using at least one of LiDAR 3 and radar 4, which are movable together with the mobile body 2. This method comprises the steps of: the environmental modeling device 1 detecting a surface present in the surrounding environment using 3D point cloud data of the surrounding environment obtained by a first measurement using LiDAR 3; calculating the front position and normal vector with respect to the detected surface; controlling the mobile body 2 to move to the front position; performing a second measurement using at least one of LiDAR 3 and radar 4, with the measurement direction aligned with the normal vector; and creating a 3D model of the surrounding environment using the 3D point cloud data obtained from the first and second measurements. By the environmental modeling device 1 performing the above method, both LiDAR 3 and radar 4 become capable of measuring specular reflection components, suppressing a decrease in 3D point cloud density and increasing the power of the reflected signal from the 3D points, thus enabling measurement with high angular resolution. By repeating this operation, it is possible to accurately model the surrounding environment.

[0057] The environmental modeling apparatus 1 according to Embodiment 1 includes a surface detection unit 13 that detects surfaces present in the surrounding environment using 3D point cloud data of the surrounding environment obtained by a first measurement using LiDAR 3, a normal vector calculation unit 14 that calculates the front position and normal vector for the surface detected by the surface detection unit 13, a movement control unit 15 that controls the moving body 2 to move it to the front position, a measurement control unit 11 that performs a second measurement using at least one of LiDAR 3 and radar 4 with the measurement direction aligned with the normal vector, and a model creation unit 17 that creates a 3D model of the surrounding environment using the 3D point cloud data obtained from the first and second measurements. As a result, the environmental modeling apparatus 1 enables both LiDAR 3 and radar 4 to measure the specular reflection component, suppressing a decrease in 3D point cloud density and increasing the power of the reflected signal from the 3D points, thus enabling measurement with high angular resolution. Furthermore, by repeating this operation, the environmental modeling apparatus 1 can accurately model the surrounding environment.

[0058] The program according to Embodiment 1 causes the computer to function as an environmental modeling device 1. This enables both the LiDAR 3 and the radar 4 to measure specular reflection components, suppressing a decrease in 3D point cloud density and allowing the computer to function as an environmental modeling device 1 that can accurately model the surrounding environment.

[0059] Embodiment 2. In the environmental modeling method according to Embodiment 2, holes, which are missing regions in the 3D point cloud data from the first measurement, are detected, and a second measurement is performed on the holes using radar, thereby interpolating the 3D point cloud data related to the holes with the second measurement. As a result, the environmental modeling method according to Embodiment 2 suppresses the decrease in 3D point cloud density and enables accurate modeling of the surrounding environment.

[0060] (Basic Configuration of the Environmental Modeling Device) Figure 5 is a block diagram showing an example of the configuration of the environmental modeling device 1A according to Embodiment 2, and shows the case when the environmental modeling device 1A is mounted on a mobile body 2. When the environmental modeling device 1A is mounted on a mobile body 2, 3D point cloud data can be acquired while moving, so a wide area of ​​the environment can be scanned in a short time. In Figure 5, the same reference numerals are used for components that are the same as in Figure 1, and redundant explanations are omitted.

[0061] For example, the environmental modeling device 1A is implemented by a computer. The memory of this computer stores programs that constitute information processing applications for realizing the functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15A, storage unit 16, model creation unit 17, and hole detection unit 18. The processor of the computer executes the information processing applications read from the memory, thereby realizing the functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15A, storage unit 16, model creation unit 17, and hole detection unit 18.

[0062] (Hole Detection Unit) The hole detection unit 18 is a gap region detection unit that determines whether a hole exists around two faces based on the degree of agreement between the normal vectors of the two faces and whether the two faces are contained in the same plane, and detects the hole in the surrounding environment based on the determination result. Here, "hole" refers to a gap in the 3D point cloud data, as described above, where 3D point cloud data was not detected in the first measurement by LiDAR3. For example, suppose the face detection unit 13 detects two faces that are not continuous due to gaps in the 3D point cloud data, and the normal vector calculation unit 14 calculates the normal vectors of these faces. This information on faces and normal vectors is output to the hole detection unit 18. The hole detection unit 18 determines whether the normal vectors calculated by the normal vector calculation unit 14 match for the two faces detected by the face detection unit 13, and whether these faces are contained in the same plane. If the normal vectors coincide and the two faces are determined to be in the same plane, the face detection unit 13 detects the areas where there are gaps in the 3D point cloud data between the two faces in the same plane as holes.

[0063] Furthermore, if the hole detection unit 18 determines that a hole exists, it calculates the position of the front of the hole based on the positions of the surfaces surrounding the hole and their normal vectors. For example, the hole detection unit 18 considers the hole region as a surface, identifies its center position, and determines the position of the measurement point S relative to these surfaces. Then, similar to Embodiment 1, the hole detection unit 18 calculates the front position P of the surface as a point moved an appropriate distance from the measurement point S in the direction of the normal vector according to the above formula (1).

[0064] To determine the normal vector of the surface containing the hole, a vector calculation is performed using the coordinate information of the 3D points that constitute the opening edge of the hole. For example, the hole detection unit 18 calculates two edge vectors from the coordinates of the 3D points that constitute the opening edge of the hole, and calculates the normal vector by taking their cross product. Then, the hole detection unit 18 normalizes the calculated normal vector to a unit vector to obtain a normal vector that represents only the direction.

[0065] (Movement Control Unit) The movement control unit 15A moves the moving body 2 based on the information of the front position of the hole and the normal vector of the surface containing the hole, calculated by the hole detection unit 18, and adjusts the attitude of the LiDAR 3 and radar 4. For example, if the moving body 2 is a vehicle (AGV) or a drone, the movement control unit 15A calculates the optimal path from the current position P0 of the moving body to the target position P. If the moving body 2 is a person, the movement control unit 15A calculates the optimal movement route to the target position P and displays an arrow or guideline on the display monitor. If AR technology is used, the direction or distance to the target position P may be overlaid within the field of view to provide intuitive movement guidance. When the moving body 2 moves to the front position P, the movement control unit 15A displays the normal vector N on the device's monitor. For example, a guideline indicating the direction of the normal vector N may be displayed on the tablet screen to allow manual adjustment.

[0066] (Environmental Modeling Method) Next, an environmental modeling method according to Embodiment 2 will be described. Figure 6 is a flowchart showing the environmental modeling method according to Embodiment 2, and illustrates a series of operations performed by the environmental modeling device 1A shown in Figure 5. The measurement control unit 11 causes the LiDAR 3 to perform a first measurement of the surrounding environment (step ST1A). The 3D point cloud data obtained from the first measurement is acquired by the data acquisition unit 12 and output to the surface detection unit 13 and the storage unit 16.

[0067] The surface detection unit 13 uses the 3D point cloud data obtained from the first measurement to detect the surfaces of objects in the surrounding environment (step ST2A). The surface information detected by the surface detection unit 13 is output to the normal vector calculation unit 14. The normal vector calculation unit 14 calculates the normal vector of the surface based on the surface information (step ST3A). When detecting a surface using a candidate normal vector, the surface detection unit 13 and the normal vector calculation unit 14 work together to calculate the normal vector and detect the surface based on the calculation result. In other words, the processes in step ST2A and step ST3A may be executed as a single process.

[0068] The hole detection unit 18 detects whether or not there is a hole in the surface (step ST4A). For example, the hole detection unit 18 determines that there is a hole if the normal vectors calculated by the normal vector calculation unit 14 match for two surfaces detected by the surface detection unit 13, and these surfaces are in the same plane. If a hole is detected in the surface (step ST4A; YES), the hole detection unit 18 calculates the front position of the hole and the normal vector of the surface containing the hole (step ST5A).

[0069] The movement control unit 15A controls the moving body drive unit 5 to move the moving body 2 to a position directly in front of the hole (step ST6A). After this, the movement control unit 15A controls the sensor drive unit 6 to adjust the measurement direction of the radar 4 to match the normal vector of the surface containing the hole. Once this adjustment is complete, the movement control unit 15A notifies the measurement control unit 11 that preparations for the second measurement are complete.

[0070] When the measurement control unit 11 receives the notification from the movement control unit 15A, it performs a second measurement, which is a remeasurement of the point cloud of the surface using the radar 4 (step ST7A). The 3D point cloud data obtained from the second measurement is acquired by the data acquisition unit 12 and output to the storage unit 16. As a result, the storage unit 16 stores the data as data in which the newly obtained 3D point cloud data from the second measurement is superimposed on the 3D point cloud data obtained from the first measurement (step ST8A). After this, the process returns to step ST4A, and the processes from step ST5A to step ST8A are executed for the newly detected hole.

[0071] On the other hand, if no holes are detected (step ST4A; NO), the model creation unit 17 creates a 3D model of the surrounding environment using the 3D point cloud data stored in the storage unit 16 (step ST9A). By the environmental modeling device 1A executing the processes from step ST1A to step ST9A, the radar 4 becomes able to measure 3D point cloud data corresponding to holes that cannot be measured even by the LiDAR3 laser beam, suppressing a decrease in 3D point cloud density and enabling accurate modeling of the surrounding environment.

[0072] In the flowchart shown in Figure 6, hole detection is performed immediately after step ST3A, but the environment modeling method according to Embodiment 2 is not limited to this. For example, the processes from step ST4A to step ST5A shown in Figure 6 and the process of step ST4 shown in Figure 2 may be performed in parallel, and depending on the distance from the current position, movement to the front position relative to the surface and movement to the front position relative to the hole may be performed, and the processes of steps ST6A and ST7A and steps ST6 and ST7 may be performed. The model creation unit 17 uses the 3D point cloud data finally obtained by these processes to create a 3D model of the surrounding environment.

[0073] Next, an overview of the environmental modeling method according to Embodiment 2 will be described. Figure 7 is a schematic diagram showing an overview of the environmental modeling method according to Embodiment 2, and shows a case where measurements are taken with an indoor space having a glass door 40 as the surrounding environment. As shown in the left side of Figure 7, LiDAR 3 irradiates the indoor space, which is the surrounding environment, with laser light A to perform the first measurement. Since the glass door 40 is made of a transparent material, the laser light A passes through it, and LiDAR 3 cannot receive the reflected light of the laser light A from the glass door 40. Therefore, although LiDAR 3 can measure 3D point cloud data from the reflected light from the surface of the wall adjacent to the glass door 40, the glass door 40 becomes a region where 3D point cloud data is missing.

[0074] In the environmental modeling device 1A, the hole detection unit 18 determines whether the normal vectors calculated by the normal vector calculation unit 14 for two surfaces corresponding to the wall adjacent to the glass door 40 detected by the surface detection unit 13 coincide and whether these surfaces are included in the same plane. In the example shown in the left diagram of Figure 7, the glass door 40 is installed on part of the same wall, so the normal vectors of the two surfaces adjacent to the glass door 40 coincide and the two surfaces are included in the same plane. For this reason, the surface detection unit 13 detects the glass door 40 as a hole where there is a gap in the 3D point cloud data.

[0075] Next, if the hole detection unit 18 determines that a hole exists, it calculates the front position P relative to the hole based on the positions and normal vectors of the surfaces surrounding the hole. The hole detection unit 18 calculates two edge vectors from the coordinates of the 3D points that constitute the opening edge of the hole, and calculates the normal vector by taking their cross product. This calculates the front position P and its normal vector relative to the glass door 40.

[0076] Next, as shown in the right-hand diagram of Figure 7, the movement control unit 15A moves the moving body 2 based on the information of the front position P of the glass door 40 and the normal vector of the surface including the glass door 40, calculated by the hole detection unit 18, to adjust the attitude of the radar 4. Once this adjustment is complete, the measurement control unit 11 performs a second measurement, which is a remeasurement of the point cloud of the surface using the radar 4. The radar 4 performs the second measurement by emitting radio waves B into the surrounding indoor environment. Since radio waves B are reflected even by the transparent glass door 40, the radar 4 can receive the reflected waves of radio waves B from the glass door 40. Furthermore, since the radar 4 performs the measurement with the measurement direction aligned with the normal vector of the glass door 40, it can measure the 3D point cloud data of the glass door 40 with high accuracy. As a result, the environmental modeling device 1A can interpolate the 3D point cloud data of objects that cannot be measured by LiDAR 3 using the radar 4.

[0077] Figure 8 is a schematic diagram showing an overview of hole detection in Embodiment 2. In Figure 8, surfaces 41 and 42 are surfaces detected from 3D point cloud data obtained in the first measurement by LiDAR3. No 3D point cloud data is detected in the region 43 between surfaces 41 and 42, and the region between surfaces 41 and 42 is separated. The hole detection unit 18 compares the normal vector 51 of surface 41 and the normal vector 52 of surface 42. If they match, the hole detection unit 18 determines whether surfaces 41 and 42 are on the same plane based on the depth information contained in the 3D point cloud data of surfaces 41 and 42. If it is determined that surfaces 41 and 42 are on the same plane, the hole detection unit 18 detects the region 43 where no 3D point cloud data is detected as a hole.

[0078] As described above, in the environmental modeling method according to Embodiment 2, the environmental modeling device 1A determines whether or not there are holes around two faces based on the degree of agreement between the normal vectors of the two faces and whether or not the two faces are contained in the same plane. This makes it possible to identify holes, which are missing regions in the 3D point cloud data from the first measurement.

[0079] In the environmental modeling method according to Embodiment 2, when the environmental modeling device 1A determines that a hole exists, it calculates the position of the front of the hole based on the positions and normal vectors of the surfaces surrounding the hole, controls the moving body 2 to move the LiDAR 3 and radar 4 to the position of the front of the hole, and performs a second measurement using radar 4 with the measurement direction aligned with the normal vector. As a result, the 3D point cloud data related to the hole can be interpolated in the second measurement, so that the decrease in 3D point cloud density is suppressed and the surrounding environment can be modeled with high accuracy.

[0080] The environmental modeling apparatus 1A according to Embodiment 2 includes a hole detection unit 18 that determines whether or not there are holes around two faces based on the degree of agreement between the normal vectors of the two faces and whether or not the two faces are contained in the same plane. As a result, the environmental modeling apparatus 1A can identify holes, which are missing regions in the 3D point cloud data in the first measurement.

[0081] In the environmental modeling device 1A according to Embodiment 2, if the hole detection unit 18 determines that a hole exists, it calculates the position of the front of the hole based on the positions of the surfaces surrounding the hole and their normal vectors. The movement control unit 15A controls the moving body 2 to move the LiDAR 3 and radar 4 to the position of the front of the hole. The measurement control unit 11 performs a second measurement using the radar 4, aligning the measurement direction with the normal vector. As a result, the environmental modeling device 1A can interpolate the 3D point cloud data related to the hole with the second measurement, thereby suppressing a decrease in 3D point cloud density and enabling accurate modeling of the surrounding environment.

[0082] The program according to Embodiment 2 causes the computer to function as an environmental modeling device 1A. This allows the computer to function as an environmental modeling device 1A that can interpolate 3D point cloud data related to holes in a second measurement.

[0083] Embodiment 3. The environmental modeling method according to Embodiment 3 detects small-area objects that result in less 3D point cloud data in the first measurement, and performs a second measurement on the small-area objects using radar, thereby interpolating the 3D point cloud data related to the small-area objects in the second measurement. As a result, the environmental modeling method according to Embodiment 3 suppresses the decrease in 3D point cloud density and enables accurate modeling of the surrounding environment.

[0084] (Basic Configuration of the Environmental Modeling Device) Figure 9 is a block diagram showing an example of the configuration of the environmental modeling device 1B according to Embodiment 3, and shows the case when the environmental modeling device 1B is mounted on a mobile body 2. When the environmental modeling device 1B is mounted on a mobile body 2, 3D point cloud data can be acquired while moving, so a wide area of ​​the environment can be scanned in a short time. In Figure 9, the same reference numerals are used for components that are the same as in Figure 1, and redundant explanations are omitted.

[0085] For example, the environmental modeling device 1B is implemented by a computer. The memory of this computer stores programs that constitute information processing applications for realizing the functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15B, storage unit 16, model creation unit 17, and rod-shaped object detection unit 19. The processor of the computer executes the information processing applications read from the memory, thereby realizing the functions of the measurement control unit 11, data acquisition unit 12, surface detection unit 13, normal vector calculation unit 14, movement control unit 15B, storage unit 16, model creation unit 17, and rod-shaped object detection unit 19.

[0086] (Rod-shaped object detection unit) The rod-shaped object detection unit 19 is a small-area object detection unit that determines whether or not a rod-shaped object exists in the surrounding environment based on the 3D point cloud data obtained in the first measurement. Here, a rod-shaped object is an example of a small-area object with a small surface area that can be a reflection point of laser light. In other words, a small-area object is any object with a small surface area, and it may be linear or spherical. For example, rod-shaped objects may be detected based on a comparison of the number of 3D point cloud data and a threshold, and if the number of 3D point cloud data exceeds the threshold, it may be determined to be a rod-shaped object. In this case, the threshold may be determined by statistically analyzing the number of 3D point cloud data obtained from measurement experiments on typical rod-shaped objects. Alternatively, a machine learning model may be used to detect rod-shaped objects, which outputs a determination result of whether or not it is a rod-shaped object when information indicating the number of 3D point cloud data and the degree of accumulation is input.

[0087] If the rod-shaped object detection unit 19 determines that a rod-shaped object is present, it calculates the position of the front of the rod-shaped object relative to its surface and its normal vector. For example, the rod-shaped object detection unit 19 identifies the center position of the surface of the rod-shaped object and determines the position of the measurement point S relative to these surfaces. Then, similar to Embodiment 1, the rod-shaped object detection unit 19 calculates the front position P of the surface as a point moved by an appropriate distance in the direction of the normal vector from the measurement point S according to the above formula (1).

[0088] Vector calculations are performed using the coordinate information of 3D points that constitute the surface of the rod-shaped object. For example, the rod-shaped object detection unit 19 calculates two edge vectors from the coordinates of 3D points that constitute the surface of the rod-shaped object, and calculates a normal vector by taking their cross product. Then, the rod-shaped object detection unit 19 normalizes the calculated normal vector to a unit vector to calculate a normal vector that represents only the direction.

[0089] (Movement Control Unit) The movement control unit 15B moves the moving body 2 based on the information of the frontal position relative to the surface of the rod-shaped object and the normal vector of that surface calculated by the rod-shaped object detection unit 19, and adjusts the attitude of the LiDAR 3 and radar 4. For example, if the moving body 2 is a vehicle (AGV) or a drone, the movement control unit 15B calculates the optimal path from the current position P0 of the moving body to the target position P. If the moving body 2 is a person, the movement control unit 15B calculates the optimal movement route to the target position P and displays an arrow or guideline on the display monitor. If AR technology is used, the direction or distance to the target position P may be overlaid within the field of view to provide intuitive movement guidance. When the moving body 2 moves to the frontal position P, the movement control unit 15B displays the normal vector N on the device's monitor. For example, a guideline indicating the direction of the normal vector N may be displayed on the tablet screen to allow manual adjustment.

[0090] (Environmental Modeling Method) Next, an environmental modeling method according to Embodiment 3 will be described. Figure 10 is a flowchart showing the environmental modeling method according to Embodiment 3, and shows a series of operations by the environmental modeling device 1B shown in Figure 9. The measurement control unit 11 causes the LiDAR 3 to perform a first measurement of the surrounding environment (step ST1B). The 3D point cloud data obtained from the first measurement is acquired by the data acquisition unit 12 and output to the surface detection unit 13 and the storage unit 16.

[0091] The surface detection unit 13 uses the 3D point cloud data obtained from the first measurement to detect the surfaces of objects in the surrounding environment (step ST2B). The surface information detected by the surface detection unit 13 is output to the normal vector calculation unit 14. The normal vector calculation unit 14 calculates the normal vector of the surface based on the surface information (step ST3B). When detecting a surface using a candidate normal vector, the surface detection unit 13 and the normal vector calculation unit 14 work together to calculate the normal vector and detect the surface based on the calculation result. In other words, the processes in steps ST2B and ST3B may be executed as a single process.

[0092] The rod-shaped object detection unit 19 detects whether or not a rod-shaped object is present (step ST4B). If a rod-shaped object is detected (step ST4B; YES), the rod-shaped object detection unit 19 calculates the frontal position of the rod-shaped object relative to its surface and the normal vector of that surface (step ST5B).

[0093] The movement control unit 15B controls the moving body drive unit 5 to move the moving body 2 to a position directly in front of the surface of the rod-shaped object (step ST6B). After this, the movement control unit 15B controls the sensor drive unit 6 to adjust the measurement direction of the radar 4 to match the normal vector of the surface containing the hole. Once this adjustment is complete, the movement control unit 15B notifies the measurement control unit 11 that preparations for the second measurement are complete.

[0094] When the measurement control unit 11 receives the notification from the movement control unit 15B, it performs a second measurement, which is a remeasurement of the point cloud of the surface using the radar 4 (step ST7B). Since the radar 4 does not have the same angular resolution as the LiDAR 3, it is relatively easy to obtain reflected waves even for rod-shaped objects. The 3D point cloud data obtained from the second measurement is acquired by the data acquisition unit 12 and output to the storage unit 16. As a result, the storage unit 16 stores the data as data in which the 3D point cloud data newly obtained from the second measurement is superimposed on the 3D point cloud data obtained from the first measurement (step ST8B). After this, the process returns to step ST4B, and the processing from step ST5B to step ST8B is executed for the newly detected rod-shaped object.

[0095] On the other hand, if the rod-shaped object is no longer detected (step ST4B; NO), the model creation unit 17 uses the 3D point cloud data stored in the storage unit 16 to create a 3D model of the surrounding environment (step ST9B). By the environmental modeling device 1B executing the processes from step ST1B to step ST9B, the radar 4 becomes able to measure 3D point cloud data corresponding to rod-shaped objects that cannot be adequately measured by the LiDAR 3 laser beam, thereby suppressing a decrease in 3D point cloud density and enabling accurate modeling of the surrounding environment.

[0096] In the flowchart shown in Figure 10, the detection of a rod-shaped object was performed immediately after step ST3B, but the environment modeling method according to Embodiment 3 is not limited to this. For example, the processes from step ST4B to step ST5B shown in Figure 10 may be performed in parallel with the process of step ST4 shown in Figure 2 and at least one of the processes of steps ST4A and ST5A shown in Figure 6. Depending on the distance from the current position, etc., movement to the front position relative to the surface and movement to the front position relative to the surface of the hole or rod-shaped object may be performed to perform the processes of steps ST6B and ST7B, and at least one of the processes of steps ST6 and ST7 and steps ST6A and ST7A. The model creation unit 17 creates a 3D model of the surrounding environment using the 3D point cloud data finally obtained by these processes.

[0097] Furthermore, Figure 9 shows an environmental modeling device 1B in which a rod-shaped object detection unit 19 is added to the configuration of Figure 1. However, the environmental modeling device 1B may also include a hole detection unit 18. This allows the environmental modeling device 1B to perform surface detection, hole detection, and rod-shaped object detection.

[0098] As described above, in the environmental modeling method according to Embodiment 3, the environmental modeling device 1B determines whether or not a rod-shaped object exists in the surrounding environment based on the 3D point cloud data obtained in the first measurement. If it determines that a rod-shaped object exists, it calculates the position of the front of the rod-shaped object relative to its surface and its normal vector. It then controls the moving body 2 to move the LiDAR 3 and radar 4 to the position of the front of the rod-shaped object relative to its surface, and as a second measurement, it uses radar 4 to perform a measurement with the measurement direction aligned to the normal vector of the surface. As a result, the 3D point cloud data related to the rod-shaped object can be interpolated in the second measurement, which suppresses a decrease in 3D point cloud density and allows for accurate modeling of the surrounding environment.

[0099] The environmental modeling device 1B according to Embodiment 3 includes a rod-shaped object detection unit 19 that determines whether or not a rod-shaped object exists in the surrounding environment based on the 3D point cloud data obtained in the first measurement. If the rod-shaped object detection unit 19 determines that a rod-shaped object exists, it calculates the position of the front of the rod-shaped object relative to its surface and its normal vector. The movement control unit 15B controls the moving body 2 to move the LiDAR 3 and radar 4 to the position of the front of the rod-shaped object relative to its surface. The measurement control unit 11 performs a second measurement using the radar 4, aligning the measurement direction with the normal vector of the surface. As a result, the environmental modeling device 1B can interpolate the 3D point cloud data related to the rod-shaped object in the second measurement, thereby suppressing a decrease in 3D point cloud density and enabling accurate modeling of the surrounding environment.

[0100] The program according to Embodiment 3 causes the computer to function as an environmental modeling device 1B. This allows the computer to function as an environmental modeling device 1B that can interpolate 3D point cloud data relating to a rod-shaped object using a second measurement.

[0101] Furthermore, it is possible to combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment.

[0102] The environmental modeling device described herein can be used, for example, to create 3D models of the surrounding environment in the fields of architecture and civil engineering.

[0103] 1, 1A, 1B Environmental modeling device, 2 Moving body, 3 LiDAR, 4 Radar, 5 Moving body drive device, 6 Sensor drive device, 11 Measurement control unit, 12 Data acquisition unit, 13 Surface detection unit, 14 Normal vector calculation unit, 15, 15A, 15B Movement control unit, 16 Storage unit, 17 Model creation unit, 18 Hole detection unit, 19 Rod-shaped object detection unit, 30 Object to be measured, 31 3D point, 40 Glass door, 41, 42 Surface, 43 Region, 51, 52 Normal vector, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104 Memory.

Claims

1. An environmental modeling method performed by an environmental modeling device that creates a three-dimensional model of the surrounding environment using three-dimensional point cloud data of the surrounding environment obtained by measurement using at least one of LiDAR and radar, which is movable with a moving object, the environmental modeling device comprising: detecting a surface present in the surrounding environment using three-dimensional point cloud data of the surrounding environment obtained by a first measurement using the LiDAR; calculating the front position and normal vector with respect to the detected surface; controlling the moving object to move to the front position; performing a second measurement using at least one of the LiDAR and radar, with the measurement direction aligned with the normal vector; and creating a three-dimensional model of the surrounding environment using the three-dimensional point cloud data obtained by the first and second measurements.

2. The environmental modeling method according to claim 1, further comprising the step of determining whether or not there are gaps in the three-dimensional point cloud data around the two faces, based on the degree of agreement between the normal vectors of the two faces and whether or not the two faces are contained in the same plane.

3. The environmental modeling method according to claim 2, comprising the steps of: when the environmental modeling device determines that the gap exists, calculating the position of the front of the gap based on the positions of the surfaces surrounding the gap and the normal vector; controlling the moving body to move it to the front position; and as a second measurement, performing a measurement using the radar with the measurement direction aligned with the normal vector.

4. The environmental modeling method according to any one of claims 1 to 3, comprising: the step of determining whether or not a small area object exists in the surrounding environment based on three-dimensional point cloud data obtained in the first measurement; if it is determined that a small area object exists, the step of calculating the front position relative to the surface of the small area object and the normal vector of the surface of the small area object; the step of controlling the moving body to move it to the front position; and the step of performing a measurement using the radar as a second measurement, with the measurement direction aligned with the normal vector.

5. An environmental modeling apparatus for creating a three-dimensional model of the surrounding environment using three-dimensional point cloud data of the surrounding environment obtained by measurement using at least one of LiDAR and radar, which is movable with a moving object, comprising: a surface detection unit for detecting surfaces present in the surrounding environment using three-dimensional point cloud data of the surrounding environment obtained by a first measurement using LiDAR; a normal vector calculation unit for calculating the front position and normal vector for the surface detected by the surface detection unit; a movement control unit for controlling the moving object to move it to the front position; a measurement control unit for performing a second measurement using at least one of LiDAR and radar, with the measurement direction aligned with the normal vector; and a model creation unit for creating a three-dimensional model of the surrounding environment using the three-dimensional point cloud data obtained by the first and second measurements.

6. The environmental modeling apparatus according to claim 5, further comprising a gap region detection unit that determines whether or not a gap region of three-dimensional point cloud data exists around the two faces based on the degree of agreement between the normal vectors of the two faces and whether or not the two faces are contained in the same plane.

7. The environmental modeling apparatus according to claim 6, wherein, if the gap area detection unit determines that a gap area exists, it calculates the position of the front of the gap area based on the positions of the surfaces surrounding the gap area and the normal vector, the movement control unit controls the moving body to move it to the front position, and the measurement control unit performs a second measurement using the radar, with the measurement direction aligned with the normal vector.

8. The environmental modeling apparatus according to any one of claims 5 to 7, comprising a small-area object detection unit that determines whether or not a small-area object exists in the surrounding environment based on three-dimensional point cloud data obtained in the first measurement, wherein if the small-area object detection unit determines that a small-area object exists, it calculates the front position of the small-area object with respect to its surface and the normal vector, the movement control unit controls the moving body to move to the front position, and the measurement control unit performs a second measurement using the radar, with the measurement direction aligned with the normal vector.

9. A program for causing a computer to function as an environmental modeling device according to any one of claims 5 to 8.

Images