Map data generation method and map data generation device

Abstract

A map data generation device (10) generates an ortho-image on the basis of a surrounding image obtained by imaging the surroundings of a vehicle (1) and the distance from the vehicle (1) to an object present around the vehicle (1), generates map data including the ortho-image, and executes, on the map data, processing for reducing the influence of a blind spot region occurring in the ortho-image.

Description

Map data generation method and map data generation device

[0001] The present disclosure relates to a map data generation method and a map data generation device.

[0002] A technique for generating an ortho image based on a surrounding image obtained by imaging the surroundings of a vehicle with an imaging device mounted on the vehicle and the distance from the vehicle to an object in the surroundings is known. An ortho image is an image equivalent to an aerial view image obtained by orthogonally projecting a surrounding image and imaging vertically from above.

[0003] For example, Patent Document 1 discloses a technique for generating an aerial view image (ortho image) based on a three-dimensional point cloud from the ground to a predetermined height and corresponding camera image data.

[0004] Japanese Unexamined Patent Application Publication No. 2013-225336

[0005] It is expected to generate map data based on the ortho image obtained as described above and utilize the map data for automatic parking, autonomous driving, etc.

[0006] However, the ortho image obtained by the technique such as Patent Document 1 has a problem that a dead zone occurs when the entire surroundings of the vehicle cannot be imaged. For example, a dead zone may occur when a part of an object shields another object.

[0007] Such a dead zone can have an adverse effect on automatic parking, autonomous driving, etc. Therefore, a technique for preferably generating map data is required.

[0008] An object of the present disclosure is to provide a map data generation method and a map data generation device that preferably generate map data in view of the above circumstances.

[0009] To achieve the above object, a map data generation method according to the present disclosure generates an ortho image based on a surrounding image obtained by imaging the surroundings of a vehicle and the distance from the vehicle to an object existing in the surroundings of the vehicle, generates map data including the ortho image, and executes a process for reducing the influence of a dead zone occurring in the ortho image on the map data.

[0010] According to this disclosure, map data can be preferably generated.

[0011] A diagram showing the overall configuration of a vehicle according to Embodiment 1 of this disclosure. A diagram showing an example of an orthoimage generated by the map data generation device according to Embodiment 1 of this disclosure. A diagram showing an example of feature points in an orthoimage generated by the map data generation device according to Embodiment 1 of this disclosure. A diagram showing an example of the hardware configuration of the map data generation device according to Embodiment 1 of this disclosure. A flowchart showing an example of map data generation processing by the map data generation device according to Embodiment 1 of this disclosure. A diagram showing the overall configuration of a vehicle according to Embodiment 2 of this disclosure. A diagram showing an example of a complementary image generated by the map data generation device according to Embodiment 2 of this disclosure. A flowchart showing an example of map data generation processing by the map data generation device according to Embodiment 2 of this disclosure.

[0012] The map data generation device according to the embodiment of this disclosure will be described below with reference to the drawings. In each drawing, the same or equivalent parts are denoted by the same reference numerals.

[0013] (Embodiment 1) Vehicle 1 according to Embodiment 1 will be described with reference to Figure 1. Vehicle 1 is, for example, a gasoline car, a diesel car, a hybrid vehicle (HV), an electric vehicle (EV), etc. Vehicle 1 comprises a map data generation device 10, an imaging unit 20, a distance measuring unit 30, a position identification unit 40, and a storage unit 50. The map data generation device 10 generates an orthomosaic image based on an image of the surroundings of vehicle 1 captured by the imaging unit 20 and the distance from vehicle 1 to surrounding objects measured by the distance measuring unit 30, and generates map data based on the generated orthomosaic image and stores it in the storage unit 50. The generated map data is used for automatic parking, automatic driving, etc.

[0014] The imaging unit 20 captures images of the area around the vehicle 1 and outputs the captured surrounding images to the map data generation device 10. The imaging unit 20 includes four optical cameras mounted in four locations on the vehicle 1, for example, the front, rear, right side, and left side. These four optical cameras can capture images of the area around the vehicle 1.

[0015] The distance measuring unit 30 measures the distance from the vehicle 1 to objects in the vicinity of the vehicle 1 and outputs distance data indicating the measured distance to the map data generation device 10. The distance measuring unit 30 includes, for example, a LiDAR (Light Detection And Ranging) mounted on the top of the vehicle 1.

[0016] The positioning unit 40 identifies the position of the vehicle 1 and outputs position data indicating the identified position to the map data generation device 10. The positioning unit 40 includes, for example, a GPS (Global Positioning System) receiver and a rotation speed sensor that measures the rotation speed of the vehicle 1's wheels. The positioning unit 40 identifies the position of the vehicle 1 based on the latitude and longitude of the vehicle 1 identified by the GPS receiver, odometry information calculated from the rotation speed of the vehicle 1's wheels, and the like.

[0017] The storage unit 50 stores the map data output by the map data generation device 10. The storage unit 50 is, for example, a flash memory installed inside the vehicle 1. The map data stored in the storage unit 50 is used for automatic parking, autonomous driving, etc. Furthermore, by copying the stored map data to the storage unit of another vehicle, it can be used not only for automatic parking and autonomous driving by vehicle 1, but also for automatic parking and autonomous driving by other vehicles.

[0018] The map data generation device 10 comprises an orthomosaic image generation unit 11, a feature point addition unit 12, a blind spot area processing unit 13, and a map data output unit 14.

[0019] The orthomosaic image generation unit 11 generates an orthomosaic image based on the surrounding image obtained by the imaging unit 20, the distance from the vehicle 1 to objects around the vehicle 1 measured by the distance measuring unit 30, and the position of the vehicle 1 identified by the position identification unit 40. More specifically, while the vehicle 1 is in motion, the imaging unit 20 continuously outputs the surrounding image, the distance measuring unit 30 continuously outputs distance data, and the position identification unit 40 continuously outputs position data. At regular intervals (for example, every second), the orthomosaic image generation unit 11 associates and stores the surrounding image, distance data, and position data at that point in time. The orthomosaic image generation unit 11 integrates the position data, surrounding image, and distance data obtained at regular intervals while the vehicle 1 is in motion to generate a single orthomosaic image. The generated orthomosaic image will include metadata related to the position. The orthomosaic image generation unit 11 generates provisional map data including the generated orthomosaic image.

[0020] The generated orthomosaic image is, for example, shown in Figure 2. The image in Figure 2 was obtained when vehicle 1 traveled on road R, entered parking lot P via side road S from road R, traveled within parking lot P, exited parking lot P via side road S back to road R, and then traveled on road R again. The black areas in Figure 2 represent the blind spots generated based on the areas that could not be captured by the imaging unit 20. For the sake of explanation, an enlarged image of the area L enclosed by the white dashed frame in Figure 2 is also included. As shown in the enlarged image of area L, in addition to the areas outside road R and parking lot P, there are also blind spots such as blind spot area B. Smaller blind spots, such as those shown in blind spot area B, are scattered within road R and parking lot P.

[0021] Refer to Figure 1 again. The feature point addition unit 12 detects feature points from the generated orthomosaic image and adds feature point data, including data about the feature points such as the location of the feature point and the feature quantity at the feature point, to the provisional map data generated by the orthomosaic image generation unit 11. Feature points are those detected using methods such as SHIFT, FAST, and Harris Corner. All of these feature point detection methods detect feature points based on brightness changes within an image within a predetermined range. Corners, edges, textures (areas with fine changes), etc., are detected as feature points using these methods. The feature points generated in this way are used for map matching, etc., performed during automatic parking and autonomous driving. Therefore, properly detecting feature points is important for the accuracy of automatic parking and autonomous driving.

[0022] However, since these feature point detection methods are based on changes in brightness within the image, blind spots can affect feature point detection. This is because blind spots are represented as black areas in the orthomosaic image, i.e., areas with zero or near-zero brightness, while areas that are not blind spots have a constant brightness in the orthomosaic image, resulting in larger changes in brightness.

[0023] For example, if feature points are detected in a portion of the orthomosaic image shown in Figure 2, and their positions are schematically represented, the result is shown in Figure 3. Note that the enlarged image of region L is omitted in Figure 3. Each white X-shape in Figure 3 represents a feature point. Because the brightness changes significantly around the blind spot region, feature points are prominently detected around the blind spot region. As a result, an unnecessarily large number of feature points are detected in the blind spot region, which can negatively affect map matching and other processes.

[0024] Refer to Figure 1 again. The blind spot processing unit 13 identifies feature points from the feature point data added to the provisional map data by the feature point addition unit 12 that have a blind spot area exceeding a predetermined percentage within a predetermined range from the feature point. The blind spot processing unit 13 detects areas in the orthomosaic image where the brightness is zero, or where the brightness is below a predetermined threshold, as blind spots. The threshold can be set by the user, and it is preferable that it be a value close to zero. In addition, the "predetermined range from the feature point" in the blind spot processing is usually the same as the "predetermined range" in the feature point detection method, but it may be set differently. The blind spot processing unit 13 removes the feature point data related to the identified feature points from the provisional map data. The points related to the removed feature point data are considered to be points that were detected as feature points due to the influence of the blind spot area. Therefore, by removing the feature point data related to such points from the provisional map data, the influence of the blind spot area occurring in the orthomosaic image can be reduced.

[0025] The map data output unit 14 includes the orthomosaic image generated by the orthomosaic image generation unit 11 and the feature point data added by the feature point addition unit 12. It outputs provisional map data, from which a portion of the feature point data has been removed by the blind spot area processing unit 13, as the final map data and stores it in the storage unit 50. This map data has feature point data related to points detected as feature points due to the influence of blind spots removed, thus reducing the influence of blind spots present in the orthomosaic image. Therefore, this map data can be suitably used in map matching in applications such as automatic parking and autonomous driving.

[0026] An example of the hardware configuration of the map data generation device 10 will be explained with reference to Figure 4. The map data generation device 10 shown in Figure 4 is implemented using a computer such as a microcontroller, smartphone, or tablet terminal.

[0027] The map data generation device 10 comprises a processor 1001, a memory 1002, an interface 1003, and a secondary storage device 1004, all of which are connected to each other via a bus 1000.

[0028] The processor 1001 is, for example, a CPU (Central Processing Unit). The processor 1001 reads the operation program stored in the secondary storage device 1004 into the memory 1002 and executes it, thereby realizing each function of the map data generation device 10.

[0029] Memory 1002 is a main memory device, for example, composed of RAM (Random Access Memory). Memory 1002 stores the operational program read by the processor 1001 from the secondary memory device 1004. Memory 1002 also functions as working memory when the processor 1001 executes the operational program.

[0030] Interface 1003 is an I / O (Input / Output) interface such as a serial port, USB (Universal Serial Bus) port, or network interface. The imaging unit 20, distance measuring unit 30, position identification unit 40, and storage unit 50 are connected to interface 1003.

[0031] The secondary storage device 1004 is, for example, flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive). The secondary storage device 1004 stores the operational programs executed by the processor 1001.

[0032] Referring to Figure 5, an example of the map data generation process by the map data generation device 10 will be explained. The process shown in Figure 5 is executed, for example, by the processor 1001 at regular intervals.

[0033] The orthomosaic image generation unit 11 of the map data generation device 10 generates an orthomosaic image based on the surrounding image output by the imaging unit 20, the distance data output by the distance measuring unit 30, and the position data output by the position identification unit 40 (step S101).

[0034] The orthomosaic image generation unit 11 generates provisional map data including the orthomosaic image generated in step S101 (step S102).

[0035] The feature point addition unit 12 of the map data generation device 10 detects feature points from the generated orthomosaic image and adds the feature point data to the provisional map data generated in step S102 (step S103).

[0036] In step S103, the blind spot area processing unit 13 of the map data generation device 10 identifies feature points from among the feature point data added to the provisional map data that have a blind spot area within a predetermined range from the feature point (step S104).

[0037] The blind spot processing unit 13 removes the feature point data related to the feature point identified in step S104 from the provisional map data to which the feature point data was added in step S103 (step S105).

[0038] In step S105, the map data output unit 14 of the map data generation device 10 outputs provisional map data from which some feature point data has been removed as the final map data (step S106). The output map data is stored in the storage unit 50. The map data generation device 10 then terminates the map data generation process.

[0039] The vehicle 1 according to Embodiment 1 has been described above. The map data generation device 10 provided in the vehicle 1 removes feature point data related to points detected as feature points due to the influence of blind spots from the map data, thereby enabling the generation of map data with reduced influence from blind spots. In other words, the map data generation device 10 can suitably generate map data. Furthermore, since the influence of blind spots is reduced in the map data generated in this way, it can be suitably utilized in map matching in automatic parking, automatic driving, etc.

[0040] (Embodiment 2) Vehicle 1 according to Embodiment 2 will be described with reference to Figure 6. However, since it is generally the same as Embodiment 1, only the differences from Embodiment 1 will be explained.

[0041] The blind spot area processing unit 13 and the feature point addition unit 12 in the second embodiment have the functions described below in addition to the same functions as those in the first embodiment.

[0042] The blind spot area processing unit 13 in the second embodiment generates a complementary image by complementing a blind spot area with respect to an ortho image including provisional map data from which some feature point data has been removed, based on a surrounding image captured by the imaging unit 20. The blind spot area processing unit 13 in the second embodiment can complement the blind spot area by, for example, replacing an image based on the surrounding image, a weighted average method, or the like.

[0043] For example, among the ortho images shown in FIG. 2, when the blind spot areas around the road R and the blind spot areas around the sidewalk S are complemented based on the surrounding image, the complementary image shown in FIG. 7 is obtained. Although such a complementary image is not correct as an aerial image, it is possible to reduce the influence of the blind spot area in the detection of feature points described later.

[0044] The blind spot area processing unit 13 in the second embodiment replaces the pre-correction ortho image including the provisional map data with the generated complementary image.

[0045] The feature point addition unit 12 in the second embodiment newly detects feature points based on the complementary image, and adds the feature point data related to the newly detected feature points to the provisional map data. It is expected that the luminance of the surrounding image used for complementing the blind spot area does not have a large difference compared to the luminance of the surrounding image in the vicinity of the blind spot area. Therefore, for example, as shown in FIG. 7, it is expected that the difference in luminance between the areas representing the road R and the sidewalk S and the luminance of the complemented area does not increase either. Therefore, when detecting feature points based on the complementary image, it is possible to prevent an unnecessarily large number of feature points from being detected compared to the case of detecting feature points based on the ortho image before complementing the blind spot area. That is, the influence of the blind spot area can be reduced. Also, compared to simply deleting some feature point data, more feature quantity data related to more feature points can be added, so it is expected that more accurate map data can be generated in map matching or the like.

[0046] Referring to FIG. 8, the map data generation process by the map data generation device 10 according to Embodiment 2 will be described. However, compared with the case of Embodiment 1 shown in FIGS. 8 and 5, since steps S201 and S202 are added between steps S105 and S106, only steps S201 and S202 will be described below.

[0047] The blind spot area processing unit 13 complements the blind spot area of the ortho-image included in the provisional map data from which some feature point data has been removed in step S105 based on the surrounding image captured by the imaging unit 20 to generate a complementary image, and replaces the ortho-image with the generated complementary image (step S201).

[0048] The feature point addition unit 12 newly detects feature points based on the complementary image, and adds the feature point data related to the newly detected feature points to the provisional map data (step S202).

[0049] As described above, the vehicle 1 according to Embodiment 2 has been described. According to the map data generation device 10 of the vehicle 1 according to Embodiment 2, after removing the feature point data related to the points detected as feature points due to the influence of the blind spot area from the map data, the blind spot area of the ortho-image is further complemented, and feature points are newly detected for the complementary image obtained by the complementation, and new feature point data is added to the map data. Therefore, according to the map data generation device 10 according to Embodiment 2, the influence of the blind spot area can be reduced as in the case of Embodiment 1, and more feature quantity data related to feature points can be added compared with the case of simply deleting some feature point data. That is, according to the map data generation device 10 according to Embodiment 2, map data can be preferably generated. And the map data generated in this way can be preferably utilized in map matching etc. in automatic parking, automatic driving, etc. because the influence of the blind spot area is reduced.

[0050] (Modifications) In each embodiment, the map data generation device 10 is provided on the vehicle 1, but the map data generation device 10 does not necessarily have to be installed on the vehicle. For example, the vehicle 1 may continuously store surrounding images, distance data, and position data in the storage unit 50, and transmit the stored surrounding images, distance data, and position data to the map data generation device 10 installed outside the vehicle, thereby generating map data. For example, the map data generation device 10 may be an external server accessible via the Internet. Alternatively, the functions of the map data generation device 10 may be divided between an in-vehicle device and an external server, and the map data generation processing may be shared between them.

[0051] In the hardware configuration shown in Figure 4, the map data generation device 10 is equipped with a secondary storage device 1004. However, the configuration is not limited to this; the secondary storage device 1004 may be located outside the map data generation device 10, and the map data generation device 10 and the secondary storage device 1004 may be connected via an interface 1003. In this configuration, removable media such as USB flash drives and memory cards can also be used as the secondary storage device 1004.

[0052] Alternatively, instead of the hardware configuration shown in Figure 4, the map data generation device 10 may be configured using a dedicated circuit that utilizes an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc. Furthermore, in the hardware configuration shown in Figure 4, some of the functions of the map data generation device 10 may be implemented, for example, by a dedicated circuit connected to interface 1003.

[0053] The program used in the map data generation device 10 can be stored and distributed on computer-readable recording media such as CD-ROM (Compact Disc Read Only Memory), DVD (Digital Versatile Disc), USB flash drive, memory card, or HDD. By installing this program on a specific or general-purpose computer, that computer can function as the map data generation device 10.

[0054] Alternatively, the aforementioned program may be stored in a storage device owned by another server on the Internet, and the program may be downloaded from that server.

[0055] This disclosure allows for various embodiments and modifications without departing from the broad spirit and scope of this disclosure. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of this disclosure. In other words, the scope of this disclosure is indicated by the claims, not by the embodiments. Various modifications made within the scope of the claims and the equivalent significance of the disclosure are considered to be within the scope of this disclosure.

[0056] 1 Vehicle, 10 Map data generation device, 11 Orthoimage generation unit, 12 Feature point addition unit, 13 Blind spot area processing unit, 14 Map data output unit, 20 Imaging unit, 30 Distance measurement unit, 40 Location identification unit, 50 Storage unit, 1000 Bus, 1001 Processor, 1002 Memory, 1003 Interface, 1004 Secondary storage device, B Blind spot area, L Area, P Parking lot, R Road, S Side road.

Claims

1. A method for generating map data, comprising: generating an orthoimage based on an ambient image obtained by imaging the area around a vehicle and the distance from the vehicle to objects present around the vehicle; generating map data including the orthoimage; and performing a process on the map data to reduce the influence of blind spots present in the orthoimage.

2. A method for generating map data according to claim 1, comprising: detecting feature points based on the brightness in the orthomosaic image included in the map data; adding data indicating the feature points to the map data; and removing data from the map data indicating feature points in which a predetermined proportion or more of the area within a predetermined range from the feature point is a blind spot, thereby reducing the influence of blind spots occurring in the orthomosaic image.

3. A method for generating map data according to claim 2, comprising: generating a complementary image by filling in the blind spots in the orthoimage based on the surrounding image; replacing the orthoimage included in the map data with the complementary image; newly detecting feature points based on the complementary image; and adding data indicating the newly detected feature points to the map data.

4. The map data generation method according to claim 1, wherein the influence of blind spots in the orthomosaic image is reduced by supplementing the blind spots in the orthomosaic image based on the surrounding image.

5. A map data generation device comprising a processor, the processor generating an orthoimage based on an ambient image obtained by imaging the area around a vehicle and the distance from the vehicle to objects present around the vehicle, generating map data including the orthoimage, and performing processing on the map data to reduce the influence of blind spots present in the orthoimage.

Images