Object recognition method and object recognition device

Abstract

This object recognition method comprises: projecting detected point-cloud objects detected from point cloud data onto an image plane; determining whether the positions of detected image objects and the positions of the detected point-cloud objects are close to each other on the image plane; calculating, for each detected image object and detected point-cloud object positioned close to each other, a shape feature quantity indicating a feature quantity of the shape on the image plane; calculating a degree of similarity on the basis of the shape feature quantities respectively calculated for the detected image object and the detected point-cloud object; and determining that the detected image object and the detected point-cloud object are the same if the degree of similarity is greater than a threshold.

Description

Object recognition method and object recognition device

[0001] The present invention relates to an object recognition method and an object recognition device.

[0002] Patent Document 1 discloses an image processing device. The image processing device acquires 3D point cloud data indicating a 3D point cloud of an object and acquires 2D label data corresponding to a 2D image of the object. The image processing device associates the 2D label data with the 3D point cloud data and converts the 2D label data into 3D point cloud label data indicating the label of the 3D point cloud.

[0003] International Publication No. 2020 / 179065

[0004] However, the generation timing of the image and the generation timing of the point cloud data may be misaligned. For example, assume that the generation period of the image and the point cloud data is 100 ms and the object is moving at a speed of 60 km / h. In this case, when the generation periods of the image and the point cloud data are misaligned, the maximum deviation amount between the object detected from the image and the object detected from the point cloud data is 0.8 m. Due to this deviation of the objects, it is impossible to accurately perform the association between the objects, and there is a problem that the recognition accuracy of the objects is reduced.

[0005] The present invention has been made in view of the above problems, and an object thereof is to provide an object recognition method and an object recognition device capable of accurately recognizing an object by accurately performing the association between the objects.

[0006] An object recognition method according to an aspect of the present invention projects a point cloud detected object detected from point cloud data onto an image plane, determines whether the position of the image detected object and the position of the point cloud detected object are close on the image plane, calculates shape feature amounts indicating the shape features on the image plane for the image detected object and the point cloud detected object whose positions are close to each other, calculates a similarity based on the shape feature amounts calculated for the image detected object and the point cloud detected object respectively, and when the similarity is greater than a threshold value, determines that the image detected object and the point cloud detected object are the same object.

[0007] According to the present invention, objects can be accurately matched to each other, thus enabling accurate recognition of objects.

[0008] Figure 1 is a block diagram showing the configuration of the object recognition device according to this embodiment. Figure 2 is a flowchart showing the main parts of the object recognition method according to this embodiment. Figure 3 is a schematic diagram showing an image-detected object on the image plane and a point cloud-detected object projected onto the image plane.

[0009] The object recognition device and object recognition method according to this embodiment will be described below with reference to the drawings. As shown in Figure 1, the object recognition device 1 recognizes objects based on images of objects around the vehicle and point cloud data relating to the positions of objects around the vehicle.

[0010] The object recognition device 1 is mounted on a vehicle. The vehicle may be a vehicle with an autonomous driving function, a vehicle without an autonomous driving function, or a vehicle that can switch between autonomous and manual driving. The autonomous driving function is a function that assists the driver by automatically controlling all or some of the vehicle control functions, such as steering control, braking force control, and driving force control. In this embodiment, the object recognition device 1 is described as being mounted on a vehicle with an autonomous driving function. The object recognition device 1 may perform vehicle control such as steering control, braking force control, and driving force control based on the recognition result (position, shape, and type of the object). This can realize highly accurate autonomous driving.

[0011] The object recognition device 1 comprises a camera 10, a distance measuring sensor 15, and a controller 20.

[0012] Camera 10 captures images of objects around the vehicle and generates images. Camera 10 is a digital camera equipped with a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 is mounted on the vehicle and generates images of objects in front of the vehicle. The images generated by camera 10 may be color images or monochrome images. Camera 10 generates images in accordance with the image acquisition timing set by the controller 20. The images generated by camera 10 are output to the controller 20.

[0013] The distance measuring sensor 15 generates point cloud data relating to the positions of multiple distance measuring points on the surface of an object in front of the vehicle. For example, the distance measuring sensor 15 measures the distance and direction to each distance measuring point by emitting electromagnetic waves around the vehicle and detecting the reflected waves from the emitted electromagnetic waves. Examples of such distance measuring sensors 15 include LIDAR (Laser Imaging Detection and Ranging) and millimeter-wave radar. The distance measuring sensor 15 is mounted on the vehicle and generates point cloud data relating to the positions of objects in front of the vehicle. The distance measuring sensor 15 generates point cloud data in accordance with the timing of point cloud data acquisition by the controller 20. The point cloud data generated by the distance measuring sensor 15 is output to the controller 20.

[0014] The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input / output unit. The microcomputer has a computer program installed that allows it to function as the object recognition device 1. By executing the computer program, the microcomputer functions as one of the multiple information processing circuits of the object recognition device 1. This embodiment shows an example of realizing the multiple information processing circuits of the object recognition device 1 using software. Of course, it is also possible to configure multiple information processing circuits for executing each of the information processing processes described below using dedicated hardware. Alternatively, multiple information processing circuits may be configured using separate hardware.

[0015] The controller 20 includes multiple information processing circuits (information processing functions), such as an image acquisition unit 21, an image object detection unit 22, a point cloud data acquisition unit 23, a point cloud object detection unit 24, a point cloud projection unit 25, an object candidate identification unit 26, a shape feature quantity calculation unit 27, a similarity calculation unit 28, and an object information integration unit 29.

[0016] The image acquisition unit 21 acquires images from the camera 10. The image acquisition unit 21 acquires images from the camera 10 at a predetermined sampling period. The image object detection unit 22 detects objects based on the images. As for object detection methods using images, known methods can be employed, such as detection methods that detect objects by comparing multiple images in a time series, methods that detect objects using template matching, and methods that detect objects using deep learning. The image object detection unit 22 recognizes the position and shape of objects on a pixel-by-pixel basis. The image object detection unit 22 also recognizes the type of object, such as cars, pedestrians, and motorcycles. Hereinafter, objects detected from images will be referred to as image-detected objects.

[0017] The point cloud data acquisition unit 23 acquires point cloud data from the distance measuring sensor 15. The point cloud data acquisition unit 23 acquires images from the distance measuring sensor 15 at a predetermined sampling period. The timing of point cloud data acquisition by the point cloud data acquisition unit 23 is controlled to coincide with the timing of image acquisition by the image acquisition unit 21. The point cloud object detection unit 24 detects objects based on the point cloud data. The point cloud object detection unit 24 groups (clusters) multiple points included in the point cloud data according to their relative distance (proximity) from each other, and recognizes the grouped set of points as an object. However, various known methods can be used as the grouping method. For example, the point cloud object detection unit 24 may detect objects using a machine learning model such as a deep neural network (DNN). Hereinafter, objects detected from the point cloud will be referred to as point cloud detected objects.

[0018] The point cloud projection unit 25 projects the point cloud detected object detected by the point cloud object detection unit 24 onto the image plane. The projection process by the point cloud projection unit 25 projects the point cloud belonging to the detected object onto the image plane. The point cloud projection unit 25 holds external parameters of the camera 10 and the distance sensor 15, such as mounting position and mounting angle. The point cloud projection unit 25 also holds internal parameters of the camera 10, such as resolution, focal length, and distortion coefficient. The point cloud projection unit 25 uses these parameters to perform the projection process.

[0019] The object candidate identification unit 26 determines whether the position of the image-detected object and the position of the point-cloud-detected object are close on the image plane. The object candidate identification unit 26 determines objects whose positions are close as corresponding object candidates. As the position of the object, the center position of a rectangle encompassing the object region on the image can be used. The shape feature amount calculation unit 27 calculates shape feature amounts that indicate the shape features on the image plane for the image-detected object and the point-cloud-detected object determined to be corresponding object candidates. The similarity calculation unit 28 calculates the similarity based on the shape feature amounts calculated for the image-detected object and the point-cloud-detected object, respectively. The object information integration unit 29 determines that the image-detected object and the point-cloud-detected object determined to be corresponding object candidates are the same object if the similarity is greater than a threshold. The object information integration unit 29 integrates the information of the image-detected object and the point-cloud-detected object and outputs it. For example, the object information integration unit 29 generates information for objects determined to be the same, using the information of the point-cloud-detected object for position and shape, and the information of the image-detected object for type.

[0020] Figure 2 is a flowchart illustrating the object recognition method according to this embodiment. The processes shown in this flowchart are executed by the controller 20.

[0021] Prior to the process shown in the flowchart of Figure 2, the image acquisition unit 21 acquires an image from the camera 10, and the image object detection unit 22 detects one or more image objects. Similarly, the point cloud data acquisition unit 23 acquires point cloud data from the distance measuring sensor 15, and the point cloud object detection unit 24 detects one or more point cloud objects. The point cloud projection unit 25 projects the one or more point cloud objects detected by the point cloud object detection unit 24 onto the image plane.

[0022] Figure 3 schematically shows an image-detected object on the image plane and a point-cloud-detected object projected onto the image plane. In Figure 3, image-detected object 100 and point-cloud-detected object 101 represent other vehicles located in front of the vehicle. Also, image-detected object 110 and point-cloud-detected object 111 represent a motorcycle located in front of the vehicle. As described above, the timing of point-cloud data generation by the distance sensor 15 is controlled to match the timing of image generation by the camera 10, but in reality, a discrepancy may occur between the two. In Figure 3, due to such a discrepancy in generation timing, image-detected objects 100 and 110 are shown in positions shifted relative to point-cloud-detected objects 101 and 111.

[0023] In performing this projection process, the point cloud projection unit 25 may perform, for example, the following processes. Specifically, the point cloud projection unit 25 estimates the vehicle's speed and angular velocity. Based on the vehicle's speed and angular velocity, the point cloud projection unit 25 corrects the relative positional shift of the point cloud detected object relative to the image detected object, which is caused by the difference between the image generation timing by the camera 10 and the point cloud data generation timing by the distance measuring sensor 15. The image generation timing by the camera 10 and the point cloud data generation timing by the distance measuring sensor 15 can be obtained as information from the camera 10 and the distance measuring sensor 15. Then, the point cloud projection unit 25 projects the corrected point cloud detected object onto the image plane. This allows the projection process to be performed while suppressing the relative positional shift of the point cloud detected object relative to the image detected object. In the above example, the point cloud projection unit 25 estimated the vehicle's speed and angular velocity. However, the point cloud projection unit 25 may also estimate the object's speed and angular velocity relative to the vehicle and correct the positional shift based on the object's speed and angular velocity.

[0024] As shown in Figure 2, the object candidate identification unit 26 selects one image detection object to be processed from among one or more image detection objects (S10). The object candidate identification unit 26 determines whether or not one or more point cloud detection objects corresponding to the selected image detection object exist on the image plane (S11). Point cloud detection objects corresponding to the selected image detection object mean point cloud detection objects that are close in position to the selected image detection object. In other words, the object candidate identification unit 26 determines whether or not one or more point cloud detection objects that are close in position to the selected image detection object exist. In this process, the position of each object can be the center position of a rectangle that encloses the object region on the image plane, for example. Furthermore, the determination of whether or not objects are close to each other is made by whether or not the distance between the center positions of the objects is less than or equal to a preset distance threshold.

[0025] If there is no point cloud detection object corresponding to the selected image detection object (S11: NO), it is considered that the selected image detection object and the point cloud detection object are separate objects. Therefore, the object candidate identification unit 26 determines that there are no identical objects (S12).

[0026] If there is one or more point cloud detection objects corresponding to the selected image detection object (S11: YES), the object candidate identification unit 26 determines that the selected image detection object and the one or more point cloud detection objects corresponding to it are corresponding object candidates. The object candidate identification unit 26 then determines whether or not there are multiple point cloud detection objects corresponding to the selected image detection object on the image plane (S13). If there is only one point cloud detection object corresponding to the selected image detection object (S13: NO), the object candidate identification unit 26 determines that the selected image detection object and the one point cloud detection object corresponding to it are the same object (S14).

[0027] If there are multiple point cloud detection objects corresponding to the selected image detection object (S13: YES), the object candidate identification unit 26 performs the processing from step S15 onwards. Hereinafter, each of the multiple corresponding point cloud detection objects will be referred to as a corresponding point cloud detection object. First, the object candidate identification unit 26 determines whether there are any unprocessed corresponding point cloud detection objects among the multiple corresponding point cloud detection objects that have not undergone the processing from steps S17 to S19 (S16). If there are unprocessed corresponding point cloud detection objects (S16: YES), the object candidate identification unit 26 performs the processing from step S17 onwards.

[0028] First, the object candidate identification unit 26 selects one corresponding point cloud detection object to be processed from among the unprocessed corresponding point cloud detection objects (S17). The shape feature calculation unit 27 calculates shape features that represent the shape features on the image plane for the image detection object selected in step S10 and the corresponding point cloud detection object selected in step S17. Methods for calculating shape features include calculating features based on the distribution of points inside the object region, such as the central moment, and calculating features based on the contour shape of the object after identifying its contour shape. These methods can represent the features as vectors with any number of elements.

[0029] The similarity calculation unit 28 calculates the similarity based on the shape features calculated for the image-detected object and the corresponding point cloud-detected object (S19). The similarity value increases as the shape features of each object approximate each other, that is, as the shape of the image-detected object on the image plane and the shape of the corresponding point cloud-detected object are closer. The similarity calculation unit 28 calculates the similarity using methods such as the Euclidean distance method and Earthmover's distance. For example, with Euclidean distance, the similarity increases as the distance between the two points represented by the two feature vectors of the image-detected object and the corresponding point cloud-detected object is closer. The similarity calculation unit 28 stores the similarity in association with the corresponding point cloud-detected object selected in step S17 (S19).

[0030] The object candidate identification unit 26 repeats the processing from steps S17 to S19 for multiple corresponding point cloud detection objects. When there are no more unprocessed corresponding point cloud detection objects (S16: NO), the object candidate identification unit 26 performs the processing in step S20. Specifically, the object candidate identification unit 26 determines whether there are any of the multiple corresponding point cloud detection objects whose similarity is greater than a threshold (S20). If there are no point cloud detection objects whose similarity is greater than a threshold (S20: NO), the object candidate identification unit 26 determines that there are no identical objects (S12). On the other hand, if there are point cloud detection objects whose similarity is greater than a threshold (S20: YES), the object candidate identification unit 26 performs the processing in step S21. The object candidate identification unit 26 determines that the corresponding point cloud detection object with the highest similarity is the same object as the selected image detection object (S21).

[0031] The threshold used in the processing of step S20 can be a pre-set fixed value. However, the object candidate identification unit 26 may change the threshold according to the attributes of the image-detected object selected in step S10. The attributes of the image-detected object are its size, shape, and distance from the vehicle. For example, in the case of small objects, objects with simple shapes, or objects located far from the vehicle, it may be difficult to recognize the difference in shape between the image-detected object and the point cloud-detected object. Therefore, the object candidate identification unit 26 increases the threshold in the case of small objects, objects with simple shapes, or objects located far from the vehicle. This makes it possible to suppress the misrecognition of objects of different shapes as the same object. Note that the size and shape of an object may be considered as the type of object.

[0032] Once objects are associated with each other in this manner, the object information integration unit 29 generates information for the objects determined to be identical, using the point cloud detection object information for position and shape, and the image detection object information for type. Note that the flowchart shown in Figure 2 shows the processing for one image detection object. Therefore, if the image object detection unit 22 detects multiple image detection objects, the processing shown in the flowchart in Figure 2 is performed for each of the detected image detection objects. Furthermore, the flowchart in Figure 2 first selects an image detection object and then evaluates the correspondence between this image detection object and multiple point cloud detection objects. However, the object recognition method according to this embodiment may also select a point cloud detection object and then evaluate the correspondence between this point cloud detection object and multiple image detection objects.

[0033] Thus, in this embodiment, the object recognition method includes projecting the point cloud detected object detected from the point cloud data onto an image plane, determining whether the positions of the image detected object and the point cloud detected object are close on the image plane, and calculating shape features for the image detected object and the point cloud detected object that are close to each other. Furthermore, the object recognition method includes calculating a similarity based on the shape features calculated for the image detected object and the point cloud detected object, determining that the image detected object and the point cloud detected object are the same object if the similarity is greater than a threshold, and generating object information using the information of the point cloud detected object for the object's position and the information of the image detected object for the object's type.

[0034] This method determines the identity of an image-detected object and a point-cloud-detected object based on the similarity between their shapes. Therefore, even if the positions of the image-detected object and the point-cloud-detected object are misaligned, they can be accurately identified as the same object. Consequently, accurate correspondence between objects can be achieved, enabling highly accurate object recognition.

[0035] The object recognition method of this embodiment may calculate shape features based on the distribution of points located inside the object region on the image plane. This method eliminates the need to calculate the object's contour shape, thus allowing shape features to be calculated with less computation.

[0036] The object recognition method of this embodiment may calculate shape features based on the contour shape of the object on the image plane. This method allows for the calculation of shape features without being affected by biases in the distribution of points inside the object region.

[0037] In the object recognition method of this embodiment, the process of projecting the point cloud detected object onto the image plane may correct the positional shift of the point cloud detected object caused by the difference between the timing of image generation by the camera 10 and the timing of point cloud data generation by the distance measuring sensor 15, based on the vehicle's speed and angular velocity, and project the corrected point cloud detected object onto the image plane. According to this method, the positional shift between the image detected object and the point cloud detected object due to the vehicle's movement can be corrected and then the point cloud detected object can be projected onto the image plane. This allows for accurate correspondence between the image detected object and the point cloud detected object.

[0038] In the object recognition method of this embodiment, the process of projecting the point cloud detected object onto the image plane may correct the positional shift of the point cloud detected object caused by the difference between the timing of image generation by the camera 10 and the timing of point cloud data generation by the distance measuring sensor 15, based on the velocity and angular velocity of the object, and project the corrected point cloud detected object onto the image plane. According to this method, the positional shift between the image detected object and the point cloud detected object due to the movement of the object can be corrected and then the point cloud detected object can be projected onto the image plane. This makes it possible to accurately associate the image detected object with the point cloud detected object.

[0039] In the object recognition method of this embodiment, when there is one image-detected object and multiple point-cloud-detected objects that are close to each other, or when there is one point-cloud-detected object and multiple image-detected objects that are close to each other, the process of calculating shape features is performed. Since it is not necessary to perform shape feature calculations for all combinations of image-detected objects and point-cloud-detected objects, the processing load can be reduced.

[0040] In the object recognition method of this embodiment, the combination of an image-detected object and a point-cloud-detected object whose similarity is greater than a threshold and whose similarity is greatest is determined to be the same object. This method allows for accurate identification of identical objects.

[0041] In the object recognition method of this embodiment, the similarity can be calculated using the Euclidean distance method. With this method, the similarity can be calculated with relatively simple calculations, and therefore the similarity of shape features can be calculated with less computation.

[0042] In the object recognition method of this embodiment, the similarity may be calculated using Earth mover's distance. Since the similarity can be calculated by considering the distance between the elements of the feature quantities, the similarity between shape features can be calculated with greater accuracy.

[0043] Not limited to the object recognition method described above, the object recognition device 1 described in this embodiment also constitutes a part of the present invention. This object recognition device includes a camera 10 mounted on a vehicle that outputs images of objects around the vehicle, a distance measuring sensor 15 mounted on a vehicle that outputs point cloud data relating to the position of objects, and a controller 20 that recognizes objects based on the images and point cloud data. This controller executes the object recognition method described above. According to this object recognition device 1, the identity of an image-detected object and a point cloud-detected object is determined based on the similarity between the shape of the image-detected object and the shape of the point cloud-detected object. Therefore, even if the position of the image-detected object and the position of the point cloud-detected object are different, the same object can be accurately identified. Consequently, accurate correspondence between objects can be performed, and objects can be recognized with high precision.

[0044] As described above, the content of the present disclosure has been explained in accordance with the embodiments. However, the present disclosure is not limited to these descriptions, and various modifications and improvements are possible.

[0045] 1: Object recognition device, 10: Camera, 15: Distance measuring sensor, 20: Controller, 21: Image acquisition unit, 22: Image object detection unit, 23: Point cloud data acquisition unit, 24: Point cloud object detection unit, 25: Point cloud projection unit, 26: Object candidate identification unit, 27: Shape feature quantity calculation unit, 28: Similarity calculation unit, 29: Object information integration unit

Claims

1. An object recognition method performed by a controller that recognizes an object based on an image of an object around a vehicle captured by a camera mounted on the vehicle and point cloud data relating to the position of the object measured by a distance measuring sensor mounted on the vehicle, comprising: detecting the object based on the image; detecting the object based on the point cloud data; projecting the point cloud detected object, which is the object detected from the point cloud data, onto an image plane; determining whether the position of the image detected object, which is the object detected from the image, and the position of the point cloud detected object are close on the image plane; calculating shape feature quantities that indicate the shape feature quantities on the image plane for the image detected object and the point cloud detected object, which are close to each other; calculating a similarity based on the shape feature quantities calculated for the image detected object and the point cloud detected object, respectively; determining that the image detected object and the point cloud detected object are the same object if the similarity is greater than a threshold; and generating information about the object using the information of the point cloud detected object for the position of the object and the information of the image detected object for the type of object.

2. The object recognition method according to claim 1, wherein the shape feature quantity is calculated based on the distribution of points located inside the object region on the image plane.

3. The object recognition method according to claim 1, wherein the shape feature quantity is calculated based on the contour shape of the object on the image plane.

4. The object recognition method according to any one of claims 1 to 3, wherein the process of projecting the point cloud detected object onto the image plane includes estimating the vehicle's speed and angular velocity based on information from a vehicle state sensor that detects the vehicle's state, correcting the relative positional shift of the point cloud detected object with respect to the image detected object caused by the difference between the timing of image generation by the camera and the timing of point cloud data generation by the distance measuring sensor, and projecting the corrected point cloud detected object onto the image plane.

5. The object recognition method according to any one of claims 1 to 3, wherein the process of projecting the point cloud detected object onto the image plane includes estimating the velocity and angular velocity of the object with respect to the vehicle, correcting the relative positional shift of the point cloud detected object with respect to the image detected object caused by the difference between the timing of image generation by the camera and the timing of point cloud data generation by the distance measuring sensor, and projecting the corrected point cloud detected object onto the image plane.

6. The object recognition method according to any one of claims 1 to 5, wherein a process is performed to calculate the shape feature quantity when there is one image-detectable object and multiple point-cloud-detectable objects that are close to each other, or when there is one point-cloud-detectable object and multiple image-detectable objects that are close to each other.

7. The object recognition method according to any one of claims 1 to 6, wherein the combination of the image-detected object and the point cloud-detected object whose similarity is greater than the threshold is determined to be the same object.

8. The object recognition method according to any one of claims 1 to 7, wherein the similarity is calculated using the Euclidean distance method.

9. An object recognition method according to any one of claims 1 to 7, wherein the similarity is calculated using Earth mover's distance.

10. An object recognition device comprising: a camera mounted on a vehicle that outputs an image of an object around the vehicle; a distance measuring sensor mounted on the vehicle that outputs point cloud data relating to the position of the object; and a controller that recognizes the object based on the image and the point cloud data, wherein the controller: detects the object based on the image; detects the object based on the point cloud data; projects the point cloud detected object, which is the object detected from the point cloud data, onto an image plane; determines whether the position of the image detected object, which is the object detected from the image, and the position of the point cloud detected object are close on the image plane; calculates shape feature quantities that indicate the shape feature quantities on the image plane for the image detected object and the point cloud detected object, which are close to each other; calculates a similarity based on the shape feature quantities calculated for the image detected object and the point cloud detected object, respectively; determines that the image detected object and the point cloud detected object are the same object if the similarity is greater than a threshold; and generates information about the object using the information of the point cloud detected object for the position of the object and the information of the image detected object for the type of the object.

Images