Data processing device, data processing method, and data processing program

Abstract

A noise determination unit (106) analyzes LiDAR data obtained by a LiDAR (200) scanning a space and millimeter wave data on which sensing points sensed by a millimeter wave radar (300) scanning the space are plotted, and determines point data that is noise among a plurality of pieces of point data included in the LiDAR data. A noise removal unit (107) removes, from the LiDAR data, point data determined to be noise by the noise determination unit (106).

Description

Data processing apparatus, data processing method and data processing program

[0001] The present disclosure relates to sensor data.

[0002] Some types of processing using sensor data require the acquired sensor data to be complete without missing portions. For example, processing that uses image data acquired from a camera requires the acquired image data to be complete without missing portions. In such processing, even if the acquired sensor data has low reliability due to noise, there is a problem in that the sensor data has to be used as it is.

[0003] Patent Document 1 discloses a sensor noise removal apparatus that enables sensor data whose reliability has been reduced due to noise to be converted into sensor data in a state where no noise is generated.

[0004] Japanese Patent Application Laid-Open No. 2024-107047

[0005] In the technique of Patent Document 1, an image processing technique is used to determine blurred portions as noise portions in image data. Further, in the technique of Patent Document 1, the presence or absence of an obstacle at a noise portion is determined from sensor data obtained from a sensor other than the camera, and the noise portion is replaced. However, there are situations in which a blurred portion cannot be determined as noise only based on data from a single sensor. For example, in a bad environment such as fog, it is difficult to determine a blurred portion as noise only based on point cloud data acquired by LiDAR. Note that LiDAR is an abbreviation for Light Detection and Ranging.

[0006] In view of such circumstances, a main object of the present disclosure is to accurately remove noise included in data acquired by a sensor.

[0007] The data processing device according to this disclosure includes a noise determination unit that analyzes point cloud data obtained by a first sensor scanning space and plot data in which sensing points sensed by a second sensor, which is a sensor of a different type from the first sensor, scanning space from a scanning position adjacent to the scanning position of the first sensor, are plotted, and determines which point data among a plurality of point data included in the point cloud data is noise, and a noise removal unit that removes the point data determined to be noise by the noise determination unit from the point cloud data.

[0008] According to this disclosure, noise contained in data acquired by the sensor can be removed with high accuracy.

[0009] A diagram showing an example of the functional configuration of the filtering device according to Embodiment 1. A diagram showing an example of the hardware configuration of the filtering device according to Embodiment 1. A flowchart showing an example of the operation of the filtering device according to Embodiment 1. A diagram explaining the free space extraction operation according to Embodiment 1. A diagram explaining the noise determination operation according to Embodiment 1. A diagram showing an example of the functional configuration of the filtering device according to Embodiment 2. A flowchart showing an example of the operation of the filtering device according to Embodiment 2. A flowchart showing an example of the operation of the filtering device according to Embodiment 2.

[0010] The embodiments will be described below with reference to the drawings. In the following description of the embodiments and in the drawings, the same reference numerals indicate the same part or a corresponding part.

[0011] Embodiment 1. ***Description of Configuration*** Figure 1 shows an example of the functional configuration of the filtering processing device 100 according to this embodiment. Figure 2 shows an example of the hardware configuration of the filtering processing device 100 according to this embodiment. The filtering processing device 100 is a data processing device that processes data acquired by a sensor. The operation procedure of the filtering processing device 100 corresponds to the data processing method. The program that realizes the operation of the filtering processing device 100 corresponds to the data processing program.

[0012] The filtering processing unit 100 is connected to the LiDAR 200 and the millimeter-wave radar 300. The LiDAR 200 scans space with a light beam and obtains point cloud data showing the distance measurement results as LiDAR data. In the LiDAR data, the positions of reflected objects (sensing points) detected by the LiDAR 200 when it receives the reflected light beam are shown as point data. The LiDAR 200 corresponds to the first sensor.

[0013] The millimeter-wave radar 300 is a different type of sensor from the LiDAR 200. Like the LiDAR 200, the millimeter-wave radar 300 scans the same space using millimeter waves and obtains plot data showing the ranging results as millimeter-wave data. The millimeter-wave radar 300 is a radar in the frequency band of 30 GHz to 300 GHz. In the millimeter-wave data, the position of the reflected object (sensing point) detected by the millimeter-wave radar 300 by receiving the reflected millimeter waves is plotted. The millimeter-wave radar 300 corresponds to the second sensor.

[0014] The scanning position of the LiDAR 200 (the installation position of the LiDAR 200) and the scanning position of the millimeter-wave radar 300 (the installation position of the millimeter-wave radar 300) are adjacent. The LiDAR 200 and the millimeter-wave radar 300 are assumed to be mounted on the same mobile device. In this embodiment, it is assumed that the LiDAR 200 and the millimeter-wave radar 300 are mounted on the same vehicle. In other words, in this embodiment, the distance between the scanning position of the LiDAR 200 and the scanning position of the millimeter-wave radar 300 is within the size of the vehicle. Therefore, the scanning position of the LiDAR 200 and the scanning position of the millimeter-wave radar 300 are in approximately the same position. The LiDAR 200 and the millimeter-wave radar 300 scan, for example, the space in front of the vehicle.

[0015] The filtering processing unit 100 is a computer and includes, for example, the hardware shown in Figure 2. Specifically, the filtering processing unit 100 includes a processor 901, main memory 902, auxiliary storage 903, and a communication device 904. The filtering processing unit 100 also includes the functional components shown in Figure 1. The functional components in Figure 1 (excluding the conversion reference information storage unit 105; the same applies hereafter) are implemented, for example, by programs. The auxiliary storage unit 903 stores programs that implement the functions of these functional components. These programs are loaded from the auxiliary storage unit 903 to the main memory 902. The processor 901 then executes these programs to perform the operation of the functional components in Figure 1. Figure 2 schematically shows the state in which the processor 901 is executing a program that implements the functions of the functional components in Figure 1. The conversion reference information storage unit 105 in Figure 1 is implemented, for example, by the auxiliary storage unit 903.

[0016] In Figure 1, the LiDAR data acquisition unit 101 performs LiDAR data acquisition processing. More specifically, the LiDAR data acquisition unit 101 acquires LiDAR data from the LiDAR 200. In this embodiment, the LiDAR data acquisition unit 101 acquires point cloud data obtained by the LiDAR 200 scanning the space in front of the vehicle as LiDAR data.

[0017] The millimeter-wave data acquisition unit 102 performs millimeter-wave data acquisition processing. More specifically, the millimeter-wave data acquisition unit 102 acquires millimeter-wave data from the millimeter-wave radar 300. In this embodiment, the millimeter-wave data acquisition unit 102 acquires plot data obtained by the millimeter-wave radar 300 scanning the space in front of the vehicle as millimeter-wave data.

[0018] The LiDAR data coordinate transformation unit 103 performs LiDAR data coordinate transformation processing. More specifically, the LiDAR data coordinate transformation unit 103 performs coordinate transformation on the LiDAR data acquired from the LiDAR data acquisition unit 101.

[0019] The millimeter-wave data coordinate transformation unit 104 performs millimeter-wave data coordinate transformation processing. More specifically, the millimeter-wave data coordinate transformation unit 104 performs coordinate transformation on the millimeter-wave data acquired from the millimeter-wave data acquisition unit 102.

[0020] The LiDAR data coordinate transformation unit 103 and the millimeter-wave data coordinate transformation unit 104 perform coordinate transformations corresponding to the positional relationship between the scanning position of the LiDAR 200 and the scanning position of the millimeter-wave radar 300. In other words, the LiDAR data coordinate transformation unit 103 and the millimeter-wave data coordinate transformation unit 104 perform coordinate transformations so that the coordinate system of the LiDAR data after transformation and the coordinate system of the millimeter-wave data after transformation are the same. For example, the LiDAR data coordinate transformation unit 103 and the millimeter-wave data coordinate transformation unit 104 perform coordinate transformations between the LiDAR data and the millimeter-wave data so that the coordinate system is based on a specific position of the vehicle. Note that if the millimeter-wave data coordinate transformation unit 104 performs coordinate transformations of the millimeter-wave data to match the coordinate system of the LiDAR data, the LiDAR data coordinate transformation unit 103 can be omitted. Conversely, if the LiDAR data coordinate transformation unit 103 performs coordinate transformation of the LiDAR data to match the coordinate system of the millimeter-wave data, then the millimeter-wave data coordinate transformation unit 104 can be omitted.

[0021] The conversion reference information storage unit 105 stores conversion reference information. The conversion reference information is information that the LiDAR data coordinate transformation unit 103 and the millimeter-wave data coordinate transformation unit 104 refer to when performing coordinate transformations. The conversion reference information includes, for example, a specific position of the vehicle that serves as the reference during coordinate transformation, the scanning position and direction of the LiDAR 200, the scanning position and direction of the millimeter-wave radar 300, etc. Note that the conversion reference information can be any information that can be used for coordinate transformations by the LiDAR data coordinate transformation unit 103 and the millimeter-wave data coordinate transformation unit 104.

[0022] The noise detection unit 106 performs noise detection processing. More specifically, the noise detection unit 106 analyzes the LiDAR data after coordinate transformation and the millimeter-wave data after coordinate transformation. The noise detection unit 106 then determines which of the multiple point data points included in the LiDAR data after coordinate transformation are noise points.

[0023] The noise reduction unit 107 performs noise reduction processing. More specifically, the noise reduction unit 107 removes point data that the noise determination unit 106 has determined to be noise from the LiDAR data after coordinate transformation.

[0024] The output unit 108 performs output processing. More specifically, the output unit 108 outputs LiDAR data after coordinate transformation, from which point data (point data determined to be noise by the noise determination unit 106) has been removed by the noise removal unit 107.

[0025] ***Explanation of Operation*** Figure 3 is a flowchart showing an example of the operation of the filtering processing device 100 according to this embodiment.

[0026] In step S101, the LiDAR data acquisition unit 101 acquires LiDAR data from the LiDAR 200. The LiDAR data acquisition unit 101 outputs the LiDAR data to the LiDAR data coordinate transformation unit 103.

[0027] In parallel, in step S102, the millimeter-wave data acquisition unit 102 acquires millimeter-wave data from the millimeter-wave radar 300. The millimeter-wave data acquisition unit 102 outputs the millimeter-wave data to the millimeter-wave data coordinate transformation unit 104.

[0028] Steps S101 and S102 are performed synchronously. In other words, the LiDAR data acquired in step S101 and the millimeter-wave data acquired in step S102 are synchronized.

[0029] In step S103, the LiDAR data coordinate transformation unit 103 acquires LiDAR data from the LiDAR data acquisition unit 101. Then, the LiDAR data coordinate transformation unit 103 performs coordinate transformation of the LiDAR data by referring to the transformation reference information. The LiDAR data coordinate transformation unit 103 outputs the transformed LiDAR data to the noise determination unit 106. In the following description of the process, unless otherwise noted, "LiDAR data" refers to the transformed LiDAR data.

[0030] In step S104, the millimeter-wave data coordinate transformation unit 104 acquires millimeter-wave data from the millimeter-wave data acquisition unit 102. The millimeter-wave data acquisition unit 102 then performs coordinate transformation of the millimeter-wave data by referring to the transformation reference information. The millimeter-wave data coordinate transformation unit 104 outputs the transformed millimeter-wave data to the noise determination unit 106. In the following description of the process, unless otherwise noted, "millimeter-wave data" refers to the transformed millimeter-wave data.

[0031] In step S105, the noise detection unit 106 analyzes the millimeter-wave data and extracts free space. Free space is a subspace in the space in front of the vehicle where no objects exist. Details of the process in step S105 will be described later.

[0032] In step S106, the noise determination unit 106 determines that point data in the LiDAR data that are located in the area corresponding to the free space extracted in step S105 are noise. The noise determination unit 106 outputs the LiDAR data and noise information notifying the point data determined to be noise to the noise removal unit 107. Details of the processing in step S106 will be described later.

[0033] In step S107, the noise reduction unit 107 removes point data that has been determined to be noise by the noise determination unit 106 from the LiDAR data. In other words, the noise reduction unit 107 removes point data that has been notified as noise information from the LiDAR data. The noise reduction unit 107 outputs the LiDAR data after the point data has been removed to the output unit 108. Alternatively, the noise reduction unit 107 may re-transform the LiDAR data after coordinate transformation, after the point data has been removed, back to the original coordinate system, and output the re-transformed LiDAR data to the output unit 108.

[0034] In step S108, the output unit 108 outputs the LiDAR data, after the point data identified as noise has been removed, to a predetermined output destination. For example, the output unit 108 outputs the LiDAR data, after the point data identified as noise has been removed, to a subsequent processing stage that performs subsequent data processing using the LiDAR data.

[0035] Next, we will explain the details of the process in step S105.

[0036] Figure 4 shows an example of millimeter-wave data after coordinate transformation. In Figure 4, the millimeter-wave radar origin corresponds to the scanning position of the millimeter-wave radar 300 (the installation position of the millimeter-wave radar 300). The fan-shaped region 310 centered on the millimeter-wave radar origin shown in Figure 4 represents the space in front of the vehicle. The nine divided regions 340 within the fan-shaped region 310 correspond to divided spaces obtained by virtually dividing the space in front of the vehicle. In Figure 4, there are nine divided regions 340, but the number of divided regions 340 is any number of two or more. In other words, the number of divisions when virtually dividing space is any number of two or more. The points labeled 320 (black circle) and 330 (×) in Figure 4 represent sensing points obtained by the millimeter-wave radar 300 scanning the space in front of the vehicle, respectively. The millimeter-wave radar 300 cannot detect particles such as fog, rain, smoke, and aerosols. Therefore, the points labeled 320 and 330 are detection points obtained by sensing objects in the space in front of the vehicle that are sufficiently larger than fog, rain, smoke, aerosols, etc. The point labeled 330 represents the detection point closest to the millimeter-wave radar origin in each divided region 340 (hereinafter referred to as the nearest neighbor point). The point labeled 320 represents a detection point other than the nearest neighbor point. The noise determination unit 106 extracts the nearest neighbor point 330 for each divided region 340. Then, for each divided region 340, the noise determination unit 106 extracts a portion of the space in front of the vehicle that is closer to the millimeter-wave radar origin than the nearest neighbor point 330 as free space. In the rightmost divided region 340a of Figure 4, the portion of space from the millimeter-wave radar origin to the nearest neighbor point 330a is free space.

[0037] If the millimeter-wave data is two-dimensional data, the noise determination unit 106 may use the FOV of the millimeter-wave radar 300 to calculate the height range of the free space and define a three-dimensional free space. FOV stands for Field of View.

[0038] Next, we will explain the details of the process in step S106.

[0039] Figure 5 shows the state in which the nearest neighbor point 330 from Figure 4 is superimposed on the LiDAR data. In Figure 5, the LiDAR origin corresponds to the scanning position of the LiDAR 200 (the installation position of the LiDAR 200). The sector-shaped region 210 centered on the LiDAR origin represents the space in front of the vehicle. In other words, the sector-shaped region 210 coincides with the sector-shaped region 310 shown in Figure 4. Also, the nine divided regions 240 within the sector-shaped region 210 each coincide with the nine divided regions 340 shown in Figure 4. The points labeled 220 (black square) and 230 (black triangle) in Figure 5 represent the sensing points (sensing positions) obtained by the LiDAR 200 scanning the space in front of the vehicle, respectively. In other words, the points labeled 220 and 230 are point data included in the LiDAR data, respectively. LiDAR200 can also detect particles such as fog, rain, smoke, and aerosols. Therefore, points labeled 220 and 230 may be detection points obtained by detecting fog, rain, smoke, aerosols, etc. On the other hand, in the analysis of millimeter-wave data, there are no objects sufficiently larger than particles in the subspace identified as free space. Therefore, detection points existing in the region corresponding to free space are highly likely to be detection points of fog, rain, smoke, aerosols, etc. Point labeled 230 is point data existing in the region corresponding to free space in Figure 4 in the LiDAR data. Point labeled 220 is point data existing outside the region corresponding to free space. The noise determination unit 106 determines that the point data labeled 230, which exists in the region corresponding to free space, is noise. In the divided region 240a at the right end of Figure 5, the noise determination unit 106 determines that the point data 230a existing in the region from the LiDAR origin to the nearest neighbor point 330a is noise. On the other hand, since point data 220a is located further from the LiDAR origin than the nearest neighbor point 330a, the noise detection unit 106 does not determine it to be noise.

[0040] The noise determination unit 106 generates noise information for notifying point data determined to be noise. Then, the noise determination unit 106 outputs the generated noise information together with the LiDAR data to the noise removal unit 107. The noise information indicates, for example, the identifier of the point data determined as noise by the noise determination unit 106 and the position of the point data in the LiDAR data. As described above, the noise removal unit 107 removes the point data notified by the noise information from the LiDAR data.

[0041] ***Description of Effects of the Embodiment*** As described above, according to the present embodiment, noise included in LiDAR data can be removed with high accuracy. According to the present embodiment, particularly when point data of fog, rain, smoke, aerosol or the like is included in LiDAR data, such point data can be removed with high accuracy. Furthermore, since the LiDAR data from which noise has been removed can be provided to post-processing, malfunctions in the post-processing can be reduced.

[0042] Embodiment 2. In the present embodiment, an example will be described in which whether or not the point data determined as noise by the noise determination unit 106 is actually noise is verified. In the present embodiment, differences from Embodiment 1 will be mainly described. Note that matters not described below are the same as those in Embodiment 1.

[0043] ***Description of Configuration*** FIG. 6 shows an example of a functional configuration of a filtering processing apparatus 100 according to the present embodiment. In FIG. 6, compared with FIG. 1, a feature amount calculation unit 111, a region division unit 112, and a noise verification unit 113 are added. The functions of the feature amount calculation unit 111, the region division unit 112, and the noise verification unit 113 are also implemented by programs similarly to the LiDAR data acquisition unit 101 and the like. The programs that implement the functions of the feature amount calculation unit 111, the region division unit 112, and the noise verification unit 113 are executed by the processor 901.

[0044] In the present embodiment, the LiDAR data coordinate conversion unit 103 outputs the coordinate-converted LiDAR data to the feature amount calculation unit 111 in addition to the noise determination unit 106.

[0045] The feature quantity calculation unit 111 performs a feature quantity calculation process. More specifically, the feature quantity calculation unit 111 acquires coordinate-converted LiDAR data from the LiDAR data coordinate conversion unit 103. Then, the feature quantity calculation unit 111 calculates a feature quantity for each piece of point data included in the acquired coordinate-converted LiDAR data.

[0046] The region division unit 112 performs a region division process. More specifically, the region division unit 112 virtually divides the space ahead of the vehicle into a non-verification target region and a verification target region. In the present embodiment, the region division unit 112 designates, as the non-verification target region, the region of the travel path on which the vehicle travels in the space ahead of the vehicle. Further, the region division unit 112 designates, as the verification target region, the region outside the travel path in the space ahead of the vehicle. In this way, the region division unit 112 divides the space ahead of the vehicle into the non-verification target region and the verification target region. The region division unit 112 outputs, to the noise verification unit 113, region range definition information that defines the range of the non-verification target region and the range of the verification target region in the coordinate-converted LiDAR data.

[0047] In the present embodiment, a case is assumed where post-stage processing is vehicle control using LiDAR data. In such vehicle control, it is desirable to remove noise as much as possible in order to detect obstacles within the travel path. Since the presence of noise increases the possibility of false detection, it is desirable to remove noise. For this reason, it is not desirable to restore point data that has been determined to be noise within the travel path. On the other hand, outside the travel path, it is desirable to secure as much point data as possible for self-position estimation of the vehicle. For this reason, it is desirable to subject point data determined to be noise to verification, and restore point data that has been removed due to erroneous determination. From such a viewpoint, in the present embodiment, the region division unit 112 designates the region of the travel path as the non-verification target region, and designates the region outside the travel path as the verification target region. Note that the region division unit 112 may designate the non-verification target region and the verification target region by a method different from this.

[0048] In the present embodiment, the noise removal unit 107 outputs noise information acquired from the noise determination unit 106 to the noise verification unit 113.

[0049] The noise verification unit 113 performs noise verification processing. More specifically, the noise verification unit 113 verifies whether point data determined to be noise by the noise determination unit 106 is noise or not. The noise verification unit 113 can refer to the noise information to identify point data that has been determined to be noise by the noise determination unit 106 and removed by the noise removal unit 107. In this embodiment, the noise verification unit 113 follows the determination result of the noise determination unit 106 for point data that exists in the non-verification target area in the LiDAR data after coordinate transformation. In other words, the noise verification unit 113 does not verify whether point data that exists in the non-verification target area in the LiDAR data after coordinate transformation is noise or not. On the other hand, for point data that exists in the verification target area in the LiDAR data after coordinate transformation, the noise verification unit 113 verifies whether point data determined to be noise by the noise determination unit 106 is noise or not. The noise verification unit 113 evaluates the feature quantities of the point data that the noise determination unit 106 has determined to be noise, and verifies whether or not the point data that the noise determination unit 106 has determined to be noise is actually noise. These feature quantities are the feature quantities calculated by the feature quantity calculation unit 111. Furthermore, if the noise verification unit 113 determines, as a result of the verification, that the point data that the noise determination unit 106 has determined to be noise is not actually noise, it restores the point data that the noise determination unit 106 has determined to be noise and that has been removed by the noise removal unit 107. The noise verification unit 113 outputs the LiDAR data after the point data has been restored to the output unit 108.

[0050] The output unit 108 outputs the LiDAR data after the point data has been restored by the noise verification unit 113.

[0051] Elements other than the LiDAR data coordinate transformation unit 103, noise reduction unit 107, output unit 108, feature quantity calculation unit 111, region division unit 112, and noise verification unit 113 are the same as those shown in Figure 1. Therefore, a detailed explanation of these elements is omitted.

[0052] ***Explanation of Operation*** Figure 7 is a flowchart showing an example of the operation of the filtering processing device 100 according to this embodiment.

[0053] In parallel with steps 101 to S107 described in Embodiment 1, in step S111, the feature calculation unit 111 calculates the feature quantities of the point data of the LiDAR data. The feature calculation unit 111 obtains the coordinate-transformed LiDAR data from the LiDAR data coordinate transformation unit 103. Then, the feature calculation unit 111 calculates the feature quantities for each point data of the coordinate-transformed LiDAR data. For example, the feature calculation unit 111 calculates the statistical values ​​of the reflection intensity (maximum value, variance, mean), the trends of first and second reflections, slice features, etc., as feature quantities. The feature calculation unit 111 outputs the calculated feature quantities for each point data to the noise verification unit 113. In the following description of the process, unless otherwise noted, "LiDAR data" means the coordinate-transformed LiDAR data.

[0054] In step S112, the noise verification unit 113 verifies the point data that the noise determination unit 106 has determined to be noise. Details of step S112 will be described later with reference to Figure 8.

[0055] In step S113, the noise verification unit 113 determines whether or not there is data to be restored. Data to be restored is the data that the noise verification unit 113 determined should be restored as a result of the verification in step S112. If there is data to be restored, the process proceeds to step S114. On the other hand, if there is no data to be restored, the process proceeds to step S115. If there is no data to be restored, the noise verification unit 113 outputs the LiDAR data after the point data has been deleted by the noise removal unit 107 to the output unit 108 as is.

[0056] In step S114, 116 restores the data to be restored in LiDAR data. 116 may restore the data to be restored in LiDAR data at the same location, using any method. 116 outputs the LiDAR data after the data to be restored to the output unit 108.

[0057] If there are multiple data points to be restored, steps S113 and S114 are performed for each data point to be restored.

[0058] In step S115, the output unit 108 outputs the LiDAR data to a predetermined output destination. If step S114 is performed, the output unit 108 outputs the LiDAR data after the recovery target point data has been recovered. On the other hand, if step S114 is not performed, the output unit 108 outputs the LiDAR data after the point data has been deleted by the noise reduction unit 107.

[0059] Figure 8 is a flowchart showing the details of step S112 in Figure 7.

[0060] In step S1121, the noise verification unit 113 selects one of the point data points that has been determined to be noise. The noise verification unit 113 identifies the point data that has been determined to be noise by referring to the noise information obtained from the noise removal unit 107.

[0061] Next, in step S1122, the noise verification unit 113 determines whether the point data selected in step S1121 is located in the verification target area. The noise verification unit 113 compares the range of the verification target area indicated in the area range definition information obtained from the feature calculation unit 111 with the position of the point data indicated in the noise information to determine whether the point data selected in step S1121 is located in the verification target area. If the point data selected in step S1121 is located in the verification target area, the process proceeds to step S1123. On the other hand, if the point data selected in step S1121 is located in the non-verification target area, the process ends.

[0062] In step S1123, the noise verification unit 113 determines whether the feature quantities of the point data selected in step S1121 meet the recovery conditions. The recovery conditions are, for example, that the feature quantities exceed a threshold. If the feature quantities of the point data selected in step S1121 exceed the threshold, the noise verification unit 113 determines that the feature quantities meet the recovery conditions. The noise verification unit 113 compares the feature quantities for each point data notified by the feature quantity calculation unit 111 with the threshold. If the feature quantities of the point data selected in step S1121 meet the recovery conditions, the process proceeds to step S1124. On the other hand, if the feature quantities of the point data selected in step S1121 do not meet the recovery conditions, the process ends.

[0063] In step S1124, the noise verification unit 113 designates the point data selected in step S1121 as the point data to be restored.

[0064] If there are multiple point data points that are determined to be noise, the noise verification unit 113 performs the processing in steps S1121 to S1124 for each point data point. Once the noise verification unit 113 has performed the processing in steps S1121 to S1124 for all point data points that have been determined to be noise, the flow in Figure 8 is completed. After that, the processing from step S113 onwards in Figure 7 is performed.

[0065] ***Explanation of the Effects of the Embodiment*** In this embodiment, the filtering strength of the point cloud data can be set according to the subsequent processing. Therefore, according to this embodiment, point cloud data suitable for subsequent processing can be extracted.

[0066] Although Embodiments 1 and 2 have been described above, these two embodiments may be implemented in combination. Alternatively, one of these two embodiments may be implemented in part. Alternatively, these two embodiments may be implemented in part in combination. Furthermore, the configurations and procedures described in these two embodiments may be modified as necessary.

[0067] ***Supplementary Explanation of Hardware Configuration*** Here, we will provide a supplementary explanation of the hardware configuration of the filtering processing device 100. The processor 901 is a CPU, DSP, etc. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. The main memory 902 is RAM. RAM is an abbreviation for Random Access Memory. The auxiliary memory 903 is a ROM, flash memory, HDD, etc. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive. The communication device 904 is an electronic circuit that performs data communication processing. The communication device 904 is, for example, a communication chip or NIC. NIC stands for Network Interface Card.

[0068] The auxiliary storage device 903 also stores the OS. OS stands for Operating System. At least a portion of the OS is executed by the processor 901. While executing at least a portion of the OS, the processor 901 executes a program that realizes the functions of the functional components shown in Figure 1. By executing the OS, the processor 901 performs task management, memory management, file management, communication control, etc. In addition, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the functional components shown in Figure 1 is stored in at least one of the main memory 902, auxiliary storage device 903, registers in the processor 901, and cache memory. Furthermore, the program that realizes the functions of the functional components shown in Figure 1 may be stored on a portable recording medium such as a magnetic disk, flexible disk, optical disk, compact disk, Blu-ray® disk, or DVD. A portable recording medium containing the program that realizes the functions of the functional components shown in Figure 1 may be distributed.

[0069] Furthermore, at least one of the "parts" of the functional components shown in Figure 1 may be read as "circuit," "process," "procedure," "processing," or "circuitry." Also, the filtering processing device 100 may be implemented by a processing circuit. The processing circuit is, for example, a logic IC, GA, ASIC, or FPGA. IC stands for Integrated Circuit. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. In this case, the functional components shown in Figure 1 are each implemented as part of a processing circuit. In this specification, the higher-level concept between the processor and the processing circuit is called "processing circuitry." In other words, a processor and a processing circuit are specific examples of "processing circuits," respectively.

[0070] The various aspects of this disclosure are described below in summary as appendices. (Appendix 1) A data processing device having a noise determination unit that analyzes point cloud data obtained by a first sensor scanning space and plot data in which sensed points sensed by a second sensor, which is a sensor of a different type from the first sensor, scanning space from a scanning position adjacent to the scanning position of the first sensor, and determines that there are noisy point data among a plurality of point data included in the point cloud data, and a noise removal unit that removes the point data determined to be noise by the noise determination unit from the point cloud data. (Appendix 2) The data processing device according to Appendix 1, wherein the noise determination unit extracts a free space, which is a subspace within the space where no objects exist, by analyzing the plot data, and determines that there are noisy point data in the region corresponding to the free space in the point cloud data. (Note 3) The data processing device according to Note 2, wherein the noise determination unit extracts the sensing point closest to the scanning position of the second sensor as the nearest neighbor point for each of the plurality of divided spaces obtained by virtually dividing the space, and extracts a subspace within the space that is closer to the scanning position of the second sensor than the extracted nearest neighbor point for each divided space as the free space. (Note 4) The data processing device according to Note 1, wherein the noise determination unit analyzes the sensing result of the second sensor after performing a coordinate transformation corresponding to the positional relationship between the scanning position of the first sensor and the scanning position of the second sensor. (Note 5) The data processing device according to Note 1, further comprising a noise verification unit that verifies whether or not point data determined to be noise by the noise determination unit is noise. (Note 6) The data processing device further has a region division unit that virtually divides the space into a non-verification region and a verification region, and the noise verification unit verifies whether point data that is in the non-verification region of the point cloud data is noise according to the determination result of the noise determination unit, and whether point data that is in the verification region of the point cloud data that has been determined to be noise by the noise determination unit is noise or not.(Note 7) The data processing device according to Note 5 or 6, wherein the noise verification unit, as a result of the verification, determines that the point data determined to be noise by the noise determination unit is not noise, and restores the point data that was determined to be noise by the noise determination unit and removed by the noise removal unit. (Note 8) The data processing device according to any one of Notes 5 to 7, wherein the noise verification unit evaluates the feature quantities of the point data determined to be noise by the noise determination unit to verify whether or not the point data determined to be noise by the noise determination unit is noise. (Note 9) The data processing device according to Note 6, wherein the first sensor and the second sensor are mounted on a vehicle traveling on a road, and the area division unit designates the area of ​​the road within the space in front of the vehicle as the area not subject to verification, and designates the area outside the road within the space in front of the vehicle as the area subject to verification, thereby dividing the space in front of the vehicle into the area not subject to verification and the area subject to verification. (Note 10) The data processing device according to any one of Notes 1 to 8, wherein the noise determination unit analyzes point cloud data obtained by scanning the space with the first sensor, which is a LiDAR (Light Detection and Ranging), and plot data in which sensing points detected by scanning the space with the second sensor, which is a millimeter-wave radar, are plotted. (Note 11) A data processing method in which a computer analyzes point cloud data obtained by scanning the space with the first sensor, and plot data in which sensing points detected by scanning the space with the second sensor, which is a sensor of a different type from the first sensor, from a scanning position adjacent to the scanning position of the first sensor, and determines which point data among a plurality of point data included in the point cloud data are noise, and the computer removes the point data determined to be noise from the point cloud data.(Note 12) A data processing program that causes a computer to perform a noise determination process to determine which point data among the multiple point data included in the point cloud data is noise, and a noise removal process to remove the point data determined to be noise by the noise determination process from the point cloud data. The program analyzes point cloud data obtained by a first sensor scanning space and plot data in which sensed points detected by a second sensor, which is a different type of sensor from the first sensor, scanning space from a scanning position adjacent to the scanning position of the first sensor.

[0071] 100 Filtering processing unit, 101 LiDAR data acquisition unit, 102 Millimeter wave data acquisition unit, 103 LiDAR data coordinate transformation unit, 104 Millimeter wave data coordinate transformation unit, 105 Transformation reference information storage unit, 106 Noise determination unit, 107 Noise removal unit, 108 Output unit, 111 Feature quantity calculation unit, 112 Region division unit, 113 Noise verification unit, 200 LiDAR, 300 Millimeter wave radar, 901 Processor, 902 Main memory, 903 Auxiliary memory, 904 Communication device.

Claims

1. A data processing device comprising: a noise determination unit that analyzes point cloud data obtained by a first sensor scanning space and plot data in which sensed points detected by a second sensor, which is a different type of sensor from the first sensor, scanning space from a scanning position adjacent to the scanning position of the first sensor, and determines which point data among a plurality of point data included in the point cloud data is noise; and a noise removal unit that removes the point data determined to be noise by the noise determination unit from the point cloud data.

2. The data processing apparatus according to claim 1, wherein the noise determination unit extracts a free space, which is a subspace within the space where no objects exist, by analyzing the plot data, and determines that point data in the region corresponding to the free space in the point cloud data is noise.

3. The data processing device according to claim 2, wherein the noise determination unit extracts the nearest sensing point closest to the scanning position of the second sensor for each of the plurality of divided spaces obtained by virtually dividing the space, and extracts a subspace within the space that is closer to the scanning position of the second sensor than the extracted nearest neighbor point for each of the divided spaces as the free space.

4. The data processing device according to any one of claims 1 to 3, wherein the noise determination unit analyzes the sensing result of the second sensor after performing a coordinate transformation corresponding to the positional relationship between the scanning position of the first sensor and the scanning position of the second sensor.

5. The data processing device according to any one of claims 1 to 4, further comprising a noise verification unit that verifies whether or not point data determined to be noise by the noise determination unit is noise.

6. The data processing device further includes a region division unit that virtually divides the space into a non-verification region and a verification region, and the noise verification unit verifies whether point data located in the non-verification region of the point cloud data is noise according to the determination result of the noise determination unit, and whether point data located in the verification region of the point cloud data that has been determined to be noise by the noise determination unit is noise or not.

7. The data processing apparatus according to claim 5 or 6, wherein, as a result of the verification, the noise verification unit determines that the point data determined to be noise by the noise determination unit is not noise, and then restores the point data that was determined to be noise by the noise determination unit and removed by the noise removal unit.

8. The data processing apparatus according to any one of claims 5 to 7, wherein the noise verification unit evaluates the feature quantities of the point data determined to be noise by the noise determination unit to verify whether or not the point data determined to be noise by the noise determination unit is noise.

9. The data processing device according to any one of claims 6 to 8, wherein the first sensor and the second sensor are mounted on a vehicle traveling on a road, and the area division unit designates the area of ​​the road within the space in front of the vehicle as the non-verification area, and the area outside the road within the space in front of the vehicle as the verification area, thereby dividing the space in front of the vehicle into the non-verification area and the verification area.

10. The data processing device according to any one of claims 1 to 8, wherein the noise determination unit analyzes point cloud data obtained by scanning the space with the first sensor, which is a LiDAR (Light Detection and Ranging), and plot data in which sensing points detected by scanning the space with the second sensor, which is a millimeter-wave radar, are plotted.

11. A data processing method comprising: a computer analyzing point cloud data obtained by a first sensor scanning space and plot data in which sensed points detected by a second sensor, which is a different type of sensor from the first sensor, scanning space from a scanning position adjacent to the scanning position of the first sensor, and determining which point data among the multiple point data included in the point cloud data are noise; and the computer removing the point data determined to be noise from the point cloud data.

12. A data processing program that causes a computer to perform a noise determination process to determine which point data among the multiple point data included in the point cloud data is noise, and a noise removal process to remove the point data determined to be noise by the noise determination process from the point cloud data. The program analyzes point cloud data obtained by a first sensor scanning space and plot data in which sensed points detected by a second sensor, which is a different type of sensor from the first sensor, scanning space from a scanning position adjacent to the scanning position of the first sensor, and performs a noise removal process.

Images