Map generation device

Abstract

This map generation device, particularly for generating a 3D HD map, comprises: an ambient recording sensor device configured to acquire recorded ambient data, the ambient recording sensor device including a recording unit that includes at least one camera unit configured to record image data as recorded ambient data, and furthermore including at least one of a radar unit, a camera unit, and / or a LiDAR unit; an object detection unit configured to receive the recorded image data and perform object detection in the image data to detect an object; an environmental data calibration unit configured to receive the recorded ambient data for a time point t of a different recording unit and convert the received recorded ambient data at the time point t of the different recording unit to a common coordinate system; and a collation unit.

Description

Map generation device

[0001] This subject matter relates to devices and methods for accurately and more efficiently automatically generating high-definition (HD) three-dimensional (3D) maps.

[0002] For example, the generation of HD maps used in the context of movement plays an important role in the autonomous driving of automobiles and other vehicles, but it is complicated and the work is done manually. In particular, map labeling / annotation is performed piecemeal by a human operator of a software program, which is time-consuming and not very efficient. This takes time, is difficult to expand, expensive, and is prone to human errors during the annotation work. Furthermore, these maps are usually created to be shown in a planar shape, i.e., 2D, and do not necessarily match the actual situation.

[0003] FIG. 1 shows an exemplary process of conventional manual 2D HD map generation, starting from a very simplified model of a part of the map, mainly including lane boundaries and center lines of roads within a map cell (the leftmost schematic diagram, (a)). Then, through the annotation work, a plurality of elements such as lane arrows, crosswalks, traffic lights, stop lines, etc. are manually added in a 2D manner. This is shown in the schematic diagram on the right side of the 2D HD map (FIG. 1(b)).

[0004] Therefore, in order to better reflect real structures such as bridges, for example, it is necessary to prepare options for more efficient and accurate methods for generating HD maps, preferably in 3D.

[0005] The above problems are overcome by the solutions presented in this disclosure and claimed in the appended claims.

[0006] One aspect of the present disclosure is a map generation apparatus for generating a 3D HD map, comprising: - an ambient recording sensor device configured to acquire recorded ambient data, wherein the ambient recording sensor device includes a recording unit comprising at least one camera unit configured to record image data as recorded ambient data; - an ambient recording sensor device further comprising at least one of the following: at least one radar unit configured to record ambient radar data as recorded ambient data, at least one camera unit configured to record ambient 3D point cloud data as recorded ambient data, and / or at least one lidar unit configured to record ambient 3D point cloud data as recorded ambient data; - an object detection unit configured to receive recorded image data and perform object detection in the image data to detect objects (data); and - an environmental data calibration unit configured to receive recorded ambient data for time t of different recording units and convert the received recorded ambient data at time t of different recording units into a common coordinate system. The present invention relates to a map generation device comprising: a matching unit configured to compare a detected object with recorded ambient data at time t of different recording units in a common coordinate system, and preferably thereby create / generate an annotated 3D HD map.

[0007] In relation to the above, it should be noted that the camera unit may include all kinds of camera units, such as mono cameras and stereo cameras. Generally, a camera or camera unit may include one or more charge-coupled image sensors or other suitable image sensors as image acquisition components. Lidar ("Optical detection and ranging") units and radar ("Radio wave detection and ranging") units are well known in the art and can be used as appropriate. The conversion of image data to 3D point cloud data may include conversion processes well known in the art, which can be performed by the camera unit itself and / or by any wirelessly and / or wired connected computing unit.

[0008] Preferably, 3D point cloud data is used for matching with detected objects, such as to generate annotated maps. LiDAR data is an option when image data is not used / cannot be used to create 3D point cloud data, and radar data is an option in particular when detecting non-static objects among detected objects.

[0009] Depending on the above configuration of the ambient recording sensor device, which may include different configurations of the recording sensor device such as a camera unit only, a camera unit and a lidar unit, a camera unit and a radar unit and a lidar unit, or other configurations of the recording sensor device, the ambient data calibration unit receives different types of recorded ambient data, i.e., image data and 3D point cloud data, i.e., image data, 3D point cloud data, radar data, etc. All such different data types are received depending on which different type of recording sensor device is used.

[0010] Such recorded ambient data received from (different) recording sensor devices is preferably received repeatedly at predefined regular or irregular time intervals, converted to a common coordinate system, e.g., a world coordinate system, and the received recorded ambient data is provided in local coordinate systems from, for example, a camera unit, a lidar unit, and / or a radar unit.

[0011] Furthermore, the matching unit is capable of performing matching of detected objects from / within image data / within image data, which, for accuracy reasons, are received as 3D point cloud data at the same time and in a common coordinate system. Such matching generates an annotated 3D point cloud map, which can then be output as an annotated map by an optional annotated map generation unit. Preferably, the matching unit and the annotated map generation unit can be a single unit that performs matching and output in particular. If radar data is available, it can be further used to remove non-static objects from the detected objects (data).

[0012] The above provides a technical advantage in that data can be recorded "on the fly," meaning that such data can be collected by any moving object equipped with the device claimed herein. For example, by moving around an object such as a car, train, or bicycle, the device equipped with the recording sensor device captures information about the surroundings / environment, and the device automatically processes the recorded data to construct at least already high-precision 3D HD annotated maps without requiring manual intervention.

[0013] Preferably, the map generation device may include a configuration in which the recording unit includes at least two camera units, one of which is configured to record image data as recorded surrounding data in the local camera unit coordinate system, and the other camera unit is configured to record 3D point cloud data as recorded surrounding data in the local camera unit coordinate system.

[0014] Preferably, the map generation device may include a configuration comprising: at least one camera unit configured to record image data as recorded ambient data in a local camera unit coordinate system; and at least one lidar unit configured to record 3D point cloud data as recorded ambient data in a local lidar unit coordinate system.

[0015] Preferably, the map generation device may further include a configuration that includes at least one radar unit configured to record surrounding radar data as recorded surrounding data in a local radar unit coordinate system.

[0016] Preferably, the matching unit compares the detected object with recorded ambient data from other recording units in a common coordinate system. In a preferred option, the matching unit may also compare the data of the detected object with the ambient data of the radar unit, in other words, the matching unit may also compare the detected object at time t in the common coordinate system with radar data at time t in the common coordinate system, and then remove non-static detected objects from the detected object (data). Such data processing relating to detection and removal can be performed by data processing methods well known in the art.

[0017] Furthermore, preferably, the matching unit matches the detected object (preferably static) at time t in the common coordinate system with a 3D point cloud map as a comparison between the detected object and recorded ambient data at time t of different recording units in the common coordinate system.

[0018] Preferably, the 3D point cloud map is generated by a 3D point cloud map generation unit of a map generation device, which uses the 3D point cloud data at time t, the 3D point cloud data at time t-1, and preferably further localization data to perform a pose-finding calculation that matches the 3D point cloud data with a (preferably pre-stored) global 3D map. Pose-finding can be calculated / performed based on / using known methods. The localization data is preferably odometry data obtained from a vehicle or moving object to which the device is attached.

[0019] Preferably, data processing performed in the apparatus disclosed herein can be carried out by the processors (CPU, GPU, ASIC, etc.) of each unit and / or the general-purpose processor of the apparatus. Furthermore, data processing may be performed remotely on a server or the like, in which case the data to be processed is preferably transmitted to the server by wireless communication means between the apparatus and the server.

[0020] Preferably, the object detection unit includes a trained artificial intelligence model that has been trained to detect objects in image data.

[0021] Preferably, the environmental data calibration unit is configured to repeatedly receive recorded ambient data at the same time t from different recording units of the ambient recording sensor device (where different units mean units present in the configuration, but preferably from all present units), and the map generation device further comprises a time synchronization unit configured to synchronize the triggers of the different recording units of the ambient recording sensor device.

[0022] In another embodiment, the disclosure also includes a system comprising a map generating device relating to at least one of the claims described above, and a mobile body, preferably a vehicle, wherein the map generating device is mounted on / inside the mobile body.

[0023] In another aspect, the Disclosure provides a (computer-implemented) method for generating a map, in particular a 3D HD map, comprising: - acquiring image data at time t by a camera unit; - acquiring 3D point cloud data at time t by a camera unit and / or a lidar unit; - preferably acquiring radar data at time t by a radar unit; - performing detection processing to detect objects in the acquired image data; - performing transformation processing to convert the image data, 3D point cloud data, and radar data, if available, from a local coordinate system to a common coordinate system; - matching the detected objects with 3D point cloud data, preferably a 3D point cloud map, in the common coordinate system; and - generating (a) a 3D annotated map.

[0024] Preferably, in a further step, radar data in a common coordinate system is compared with multiple detected objects, and if a detected object is determined not to be static, that detected object is removed from the multiple detected objects. For such detection of static objects, it is preferable to use information obtained from radar data, and generally known techniques can be used for such detection of static objects and removal from object detection data.

[0025] Preferably, the 3D point cloud map is generated by comparing 3D point cloud data at time t with 3D point cloud data at a previous time, such as t-1. Preferably, the 3D point cloud map is created by finding a pose transformation that matches the point cloud at time t with a global 3D map that is generally available, for example, pre-stored in the memory of the device or a connected server.

[0026] Preferably, the timing of repeated acquisition of image data, lidar data, and / or radar data is synchronized by a trigger signal issued by a time synchronization unit, and very preferably, the time synchronization unit is the same unit / device for all recording units, so that all recording units are triggered by the same device and, consequently, by the same trigger signal, thereby ensuring synchronization.

[0027] Preferably, the (generated) 3D annotated map is uploaded to a remote server and subsequently provided to other users via wireless communication to other vehicles, for example, to use as a navigation map.

[0028] Preferably, the method and / or preferably a part thereof can be carried out by an apparatus such as that described in the embodiment relating to the map generation apparatus above.

[0029] Another embodiment may further relate to a computer program product that can be stored in memory, which, when executed by a computer, includes instructions that cause the computer to perform the aforementioned methods and / or preferably a portion thereof.

[0030] As stated above, this disclosure offers technical advantages, particularly in automatically recording data while in motion and using such data for automated processing to generate high-precision annotated / HD maps.

[0031] The following drawings are line drawings that replace the image (photograph) portions of the drawings in the original application (European Patent Application No. 25290004.8) with line drawings for clarity, and no new subject matter has been added. An example of a prior art generation process relating to a portion of a 2D map is shown. A schematic diagram of the map generation device relating to this disclosure is shown. A schematic diagram of the data flow relating to a portion of the map generation device relating to this disclosure, with preferred options, is shown. A schematic diagram of the data flow relating to a portion of the map generation device relating to this disclosure, with preferred options, is shown. A schematic diagram of the data flow relating to a portion of the map generation device relating to this disclosure is shown. A schematic diagram of a possible embodiment relating to a time trigger relating to this disclosure is shown. A schematic diagram of the data flow between the server, other vehicles, and the upload of the generated map is shown.

[0032] Figure 1 shows the process of the latest technology (conventional technology) for creating annotated maps, resulting in the generation of an annotated 2D map, with annotations performed by a human computer operator, i.e., through manual / graphic reprocessing of the data.

[0033] In contrast to such processing, this disclosure utilizes a device 100 that automatically generates 3D annotated maps.

[0034] Figure 2 schematically shows some components of the map generation device 100 according to this disclosure. The upper part of Figure 2 shows components that are primarily involved in processing image data and radar data (the latter, if available), which are provided by one or more radar units 1c and / or one or more camera units 1a. These units 1a, 1c repeatedly acquire information such as image data and radar data from the environment / surroundings / surroundings (abbreviated as "device") of the map generation device 100. Repeated data acquisition may be performed at regular or irregular intervals, which may be fixed, such as programmed intervals, or may be set by the user of the device 100 before use. The device 100 itself can be mounted / attached to different mobile bodies, preferably vehicles.

[0035] A trigger signal for repeated data acquisition can be provided by a time synchronization unit 2, which simultaneously triggers all available recording units 1a-1c and / or odometry unit 1d. Such a commonly triggered process is shown in Figure 7 as a possible example. Figure 7 shows that all recording devices 1a-1c, including odometry unit 1d, are triggered at the same time interval, for example, as shown in dt(s), so that the data acquired by each recording unit is reliably obtained from the same point in time.

[0036] Figure 2 further shows radar and image data at time t, for example, and it is clear that such processing is repeated at further time points t+1, t+2, etc., and that Figure 2 shows an exemplary procedure at exemplary time point t. The image data is provided to another unit of the device 100, which may optionally be part of an external system in which case the image data is transmitted to the other unit, for example wirelessly, processed, and returned to the device. In either case, the other unit is preferably an object detection unit 3 employing a trained machine learning model (ML) / artificial intelligence (AI) model to perform object detection in the image data. This allows the class, shape, location, and / or other information of one or more objects to be extracted from the image data via the object detection unit 3. For example, the extraction / detection may involve any part of a street, a building or structure along the street, a tree, or other objects.

[0037] Furthermore, if radar data is available, it can be used to determine whether the detected object is static or not.

[0038] The data from the different recording units 1a to 1c of the ambient recording sensor device 1 are then transformed using a calibration matrix or a corresponding method, so that the coordinate system of each recording unit, which is a local coordinate system, is transformed into a common coordinate system, such as world coordinates.

[0039] The processed data, transformed into a common coordinate system, is then input into another unit called the matching unit 5. Its function is described further below.

[0040] As further shown in Figure 2, the 3D point cloud map generation unit 7 provides the 3D point cloud map 40 to the matching unit 5. The generation / creation of the 3D point cloud map 40 is preferably performed using LiDAR-based point cloud data and / or camera-based image data provided as point cloud data, as shown in Figure 2. An overview of this process is shown at the bottom of Figure 2, where the dashed box indicates the process (the 3D point cloud map generation unit 7 may include at least all of the subunits shown in the box in Figure 2, but may include fewer subunits; for example, the LiDAR and odometry unit 1d may rather be locally isolated and communicated via wired or wireless means).

[0041] As described above, the procedure for generating the 3D point cloud map 40 uses point cloud data from two different points in time, for example, t and t-1, i.e., the actual time and a previous time such as one frame or one trigger before. Odometry data from the odometry unit 1d is also merged with the point cloud data, which allows for the discovery of pose transformations to match the point cloud at time t with the global 3D map. The results of this process, which is performed repeatedly as indicated by the arrows, are shown in the right side of the lower frame in Figure 2, i.e., the 3D point cloud map 40.

[0042] The 3D point cloud map 40 shown in Figure 2 is input to the matching unit 5. The matching unit 5 then possesses both the detected object data and the 3D point cloud map 40 as information in a common coordinate system. By comparing these, the matching unit 5 creates an annotated 3D HD map 50 and outputs it via the annotated map generation unit 6. The latter unit may also be part of the matching unit 5.

[0043] Figure 3 shows a part of the process flow included in Figure 2, focusing on the processing of data such as data acquired at time t. As described above, different recording units 1a to 1c, for example, camera 1a, LiDAR 1b, and / or radar 1c provide data. In Figure 3, camera 1a provides image data at different time points in the local camera coordinate system, in this example, at time t. The same applies to LiDAR 1b that provides 3D point cloud data at time t in the LiDAR coordinate system, and radar 1c that provides radar data in the radar coordinate system. Such (recorded environment / surrounding) data is input into the environmental data calibration unit 4 in the case of pre-processed image data for detecting objects in the image data as described above, and the data is converted into a common coordinate system such as the world coordinate system using a calibration matrix or other corresponding methods. Thereafter, the environmental data of different origins described above becomes available in the common coordinate system, and a matching process can be executed using the common coordinate environmental data. Figure 4 shows a similar process, but alternatively, the point cloud data is provided by the camera unit instead of the LiDAR unit, and the combination of both sources of images and LiDAR is also an option included in the present disclosure. In this context, these examples are non-limiting examples of the disclosure, and it should be further noted that other sources of recording units such as radar units can provide point cloud data, and these units can be used alternatively or additionally.

[0044] The matching is executed as described in relation to Figure 2 above, and Figure 5 shows the steps of a sub-matching process that particularly distinguishes moving / non-static objects from static objects. For this purpose, steps S1 to S2 of comparing the object detection result at time t with the radar data at time t are applied. If S1 indicates a moving object, they are excluded from the object detection data, or at least no further processing is performed.

[0045] Next, step 3 shows the above-described procedure steps executed in the matching unit 5, where the point cloud data and the detected object data are merged with each other. As a result, it becomes possible to create an annotated 3D HD map 50 in the unit 5 or the unit 6 (which can be a combining unit).

[0046] Furthermore, FIG. 6 shows another excerpt of the general processing shown in FIGS. 2 to 4, showing the matching between the point cloud data and the image-based object detection data (detected objects). The excerpt of the processing shown in FIG. 6 specifically shows the creation of the 3D point cloud map 40 described above.

[0047] Finally, FIG. 8 shows a possible application scenario in which the device 100 disclosed in this specification is attached to a vehicle traveling on a road and processes the environment to create a 3D HD map 50. Thereafter, this map 50 is transmitted to the server 200, and the server can provide such map data to other vehicles v1 to v3 (almost) in real time.

[0048] Therefore, the present disclosure improves the generation of a 3D map having annotations automatically generated by the device 100 described above.

[0049] Furthermore, note that the examples of the present disclosure can take the form of a completely hardware example, a completely software example (including firmware, resident software, microcode, etc.), or an example combining software and hardware aspects. Furthermore, the examples of the present disclosure may take the form of a computer program product on a computer-readable medium having computer-executable program code incorporated in a medium.

[0050] In the drawings, note that arrows can be used to represent communication, transfer, or other activities spanning two or more entities. A two-way arrow generally indicates that an activity can occur bidirectionally (e.g., a command / request in one direction and a corresponding response in the other direction, or peer-to-peer communication initiated by any entity), but depending on the situation, the activity may not necessarily occur bidirectionally.

[0051] While unidirectional arrows generally indicate one-way, or primarily one-way, activity, it should be noted that in certain situations, such directional activity may actually include bidirectional activity (e.g., a message from sender to receiver and a reply from receiver to sender, or connection establishment before forwarding and connection termination after forwarding). Therefore, the type of arrow used to represent a particular activity in a particular drawing is illustrative and should not be considered limiting.

[0052] The embodiments / examples were described above with reference to flowcharts and / or block diagrams of methods and apparatus. It will be understood that each block in the flowcharts and / or block diagrams, and / or combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer executable program code.

[0053] Computer executable program code may be provided to a processor of a general-purpose computer, a dedicated computer, or other programmable data processing device for manufacturing a particular machine, and as a result, program code executed via the processor of the computer or other programmable data processing device forms means for implementing functions / operations / outputs specified by flowcharts, block diagrams, or a series of blocks, diagrams, and / or descriptive explanations.

[0054] These computer-executable program codes may also be stored in computer-readable memory that can instruct a computer or other programmable data processing device to function in a particular way, and as a result, the program code stored in computer-readable memory may produce a product that includes instruction means for implementing functions / operations / outputs specified by flowcharts, block diagram blocks, figures, and / or descriptive explanations.

[0055] Computer-executable program code may also be loaded into a computer or other programmable data processing device and generate a computer implementation process by causing the computer or other programmable device to execute a series of operational steps, thereby providing steps for implementing functions / operations / outputs specified in a flowchart, block diagram block, diagram, and / or descriptive explanation. Alternatively, steps or operations performed by a computer program may be combined with steps or operations performed by an operator or human to execute an embodiment.

[0056] Communication networks may generally include public and / or private networks, and may include local area, wide area, metropolitan area, storage, and / or other types of networks, and may employ communication technologies including, but not limited to, analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0057] It should also be noted that devices may use communication protocols and messages (e.g., messages created, sent, received, stored, and / or processed by the device), and such messages may be transmitted by communication networks or media.

[0058] Unless otherwise required by the context, this disclosure should not be construed as being limited to any particular communication message type, communication message format, or communication protocol. Therefore, communication messages may generally include, but are not limited to, frames, packets, datagrams, user datagrams, cells, or other types of communication messages.

[0059] Unless otherwise specified in the context, references to specific communication protocols are illustrative, and alternative embodiments may, as needed, employ variations of such communication protocols (e.g., modifications or extensions to protocols that may be made from time to time), or other well-known or future-developed protocols.

[0060] While logical flows may be described herein to demonstrate various aspects, it should be noted that this disclosure should not be construed as limiting to any particular logical flow or implementation. The described logic can be divided into different logical blocks (e.g., programs, modules, functions, or subroutines) without altering the overall result.

[0061] In many cases, logical elements can be added, modified, omitted, executed in a different order, or implemented using different logical structures (e.g., logic gates, looping primitives, conditional logic, and other logical structures) without changing the overall result.

[0062] This disclosure is not limited to, but may be incorporated in many different forms, including computer program logic for use in a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general-purpose computer), programmable logic for use in a programmable logic device (e.g., a field-programmable gate array (FPGA) or other PLD), individual components, integrated circuits (e.g., application-specific integrated circuits (ASICs)), or any other means including any combination thereof. Computer program logic implementing some or all of the described functions is typically translated into a computer executable form, stored in that state on a computer-readable medium, and implemented as a set of computer program instructions executed by a microprocessor under the control of an operating system. Hardware-based logic implementing some or all of the described functions may be implemented using one or more appropriately configured FPGAs.

[0063] Computer program logic that implements all or part of the functions described herein may be embodied in a variety of forms, including but not limited to source code, computer executable, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator).

[0064] The source code may include a set of computer program instructions implemented in one of various programming languages ​​(e.g., object code, assembly language, or high-level languages ​​such as Fortran, C, C++, Java, or HTML) and be used in various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in computer executable format (e.g., via an interpreter), or the source code may be converted to computer executable format (e.g., via a translator, assembler, or compiler).

[0065] Computer executable program code for performing the operations of the embodiments of this disclosure may be written in an object-oriented, scripting, or non-scripting programming language such as Java, Perl, Smalltalk, or C++. However, computer executable program code for performing the operations of the embodiments may also be written in a conventional procedural programming language such as the C programming language or a similar programming language.

[0066] Computer program logic that implements all or part of the functions described herein may run on a single processor (e.g., simultaneously) at different times, on multiple processors at the same or different times, or under a single operating system process / thread or under different operating system processes / threads.

[0067] Therefore, the term "computer process" can generally refer to the execution of a set of computer program instructions, regardless of whether different computer processes run on the same or different processors, or whether different computer processes operate under the same or different operating system processes / threads.

[0068] Computer programs may be permanently or temporarily fixed in any form (e.g., source code, computer executable, or intermediate) to tangible storage media such as semiconductor memory devices (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), magnetic memory devices (e.g., floppy disks or fixed disks), optical memory devices (e.g., CD-ROMs), PC cards (e.g., PCMCIA cards), or other memory devices.

[0069] Computer programs may be fixed in any form to signals that can be transmitted to a computer using any of a variety of communication technologies, including but not limited to analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0070] Computer programs may be distributed in any form as removable storage media accompanied by accompanying printed materials or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto computer systems (e.g., on system ROM or fixed disks), or distributed from servers or electronic bulletin boards via communication systems (e.g., the Internet or the World Wide Web).

[0071] Hardware logic (including programmable logic for use in programmable logic devices) that implements all or part of the functions described herein may be designed using conventional manual methods, or may be designed, captured, simulated, or documented electronically using various tools such as computer-aided design (CAD), hardware description languages ​​(e.g., VHDL or AHDL), or PLD programming languages ​​(e.g., PALASM, ABEL, or CUPL).

[0072] Any suitable computer-readable medium may be used. This computer-readable medium may, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, devices, or media.

[0073] More specific examples of computer-readable media include, but are not limited to, electrical connections having one or more wires, or other tangible storage media such as portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disk read-only memory (CD-ROM), or other optical or magnetic storage devices.

[0074] The programmable logic may be permanently or temporarily fixed to a tangible storage medium such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a floppy disk or fixed disk), an optical memory device (e.g., a CD-ROM), or other memory device.

[0075] Programmable logic may be fixed to a signal that can be transmitted to a computer using any of a variety of communication technologies, including but not limited to analog technology, digital technology, optical technology, wireless technology (e.g., Bluetooth), networking technology, and internetworking technology.

[0076] The programmable logic may be distributed as a removable storage medium accompanied by accompanying printed or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., on a system ROM or fixed disk), or distributed from a server or electronic bulletin board via a communication system (e.g., the Internet or the World Wide Web). Of course, some embodiments can be implemented as a combination of both software (e.g., computer program products) and hardware. Still other embodiments can be implemented entirely as hardware or entirely as software.

[0077] While certain exemplary embodiments are described and illustrated in the accompanying drawings, it should be understood that such embodiments are illustrative and not limited to the specific structures and arrangements illustrated and described, as various other changes, combinations, omissions, modifications, and substitutions are possible in addition to those described in the paragraphs above.

[0078] Those skilled in the art will understand that various adaptations, modifications, and / or combinations of the embodiments described above are possible. Therefore, it should be understood that within the scope of the appended claims, the disclosure may be implemented in ways other than those specifically described herein. For example, unless otherwise specified, the steps of the processes described herein may be performed in an order different from that described herein, and one or more steps may be combined, separated, or performed simultaneously. Those skilled in the art will also understand that, considering the disclosure, different examples or embodiments described herein can be combined to form other examples.

[0079] 100 Map generation device, 50 3D HD map, 1 Surroundings recording sensor device, 1a-1c Recording unit, 1a Camera unit, 1b LiDAR unit, 1c Radar unit, 1d Odometry, 2 Time synchronization unit, 3 Object detection unit, 4 Environmental data calibration unit, 5 Matching unit, 6 Annotated map generation unit, 7 3D point cloud map generation unit

Claims

1. A map generation device, particularly for generating a 3D HD map, comprising: an ambient recording sensor device configured to acquire recorded ambient data, wherein the ambient recording sensor device includes a recording unit comprising at least one camera unit configured to record image data as recorded ambient data, and further comprising at least one of the following: at least one radar unit configured to record ambient radar data as recorded ambient data, at least one camera unit configured to record ambient 3D point cloud data as recorded ambient data, and / or at least one lidar unit configured to record ambient 3D point cloud data as recorded ambient data; an object detection unit configured to receive recorded image data and perform object detection in the image data to detect an object; an environmental data calibration unit configured to receive the recorded ambient data for time t of a different recording unit and convert the received recorded ambient data at time t of the different recording unit into a common coordinate system; and a matching unit configured to match a detected object with the recorded ambient data at time t of the different recording unit in the common coordinate system.

2. The map generation apparatus according to claim 1, wherein the recording unit includes at least two camera units, one of which is configured to record image data as recorded surrounding data in the local camera unit coordinate system, and the other camera unit is configured to record 3D point cloud data as recorded surrounding data in the local camera unit coordinate system.

3. The map generation apparatus according to claim 1 or 2, wherein the recording unit includes at least one camera unit configured to record image data as recorded ambient data in the local camera unit coordinate system, and at least one lidar unit configured to record 3D point cloud data as recorded ambient data in the local lidar unit coordinate system.

4. The map generation apparatus according to at least one of claims 1 to 3, preferably claim 2 or 3, wherein the recording unit further comprises at least one radar unit configured to record the surrounding radar data as recorded surrounding data in a local radar unit coordinate system.

5. The map generation apparatus according to at least one of claims 1 to 4, wherein the matching unit compares the detected object with the recorded ambient data from another recording unit in the common coordinate system, preferably, the matching unit compares the detected object at time t in the common coordinate system with radar data at time t in the common coordinate system, removes non-static detected objects from the detected object, and matches the static detected object at time t in the common coordinate system with a 3D point cloud map.

6. The map generation apparatus according to claim 5, wherein the 3D point cloud map is generated by a 3D point cloud map generation unit, and this 3D point cloud map generation unit performs a calculation to find a pose that matches the 3D point cloud data with a global 3D map using the 3D point cloud data at time t, the 3D point cloud data at time t-1, and position identification data.

7. The map generation apparatus according to at least one of claims 1 to 6, wherein the object detection unit includes a trained artificial intelligence model trained to detect objects in image data.

8. The map generation device according to at least one of claims 1 to 7, wherein the environmental data calibration unit is configured to repeatedly receive the recorded ambient data from the different recording units of the ambient recording sensor device at the same time t, and the map generation device further comprises a time synchronization unit configured to synchronize the triggers of the different recording units of the ambient recording sensor device.

9. A system comprising a map generating device and a vehicle according to at least one of claims 1 to 8, wherein the map generating device is mounted on / inside the vehicle.

10. A computer implementation method for generating a map, in particular a 3D HD map, comprising: acquiring image data at time t by a camera unit; acquiring 3D point cloud data at time t by a camera unit and / or a lidar unit; acquiring radar data at time t by a radar unit; performing detection processing for detecting objects in the acquired image data; performing transformation processing to convert the image data, the 3D point cloud data, and the radar data from a local coordinate system to a common coordinate system; and matching the detected objects with the 3D point cloud data in the common coordinate system.

11. The method according to claim 10, wherein in a further step, the radar data in the common coordinate system is compared with the plurality of detected objects, and if a detected object is determined not to be static, the detected object is removed from the plurality of detected objects.

12. The method according to at least one of claim 10 or 11, wherein the 3D point cloud map is generated by matching 3D point cloud data at time t and time t-1.

13. The method according to at least one of claims 10 to 12, wherein the time points for repeatedly acquiring image data, lidar data, and / or radar data are synchronized by trigger signals issued by a time synchronization unit.

14. The method according to at least one of claims 10 to 13, wherein the 3D annotated map is uploaded to a remote server.

15. The method according to at least one of claims 10 to 14, performed by the apparatus according to at least one of claims 1 to 8.

Images