Method for calibrating a lidar sensor and a camera sensor of a vehicle

Abstract

The invention relates to a computer-implemented method (100) for calibrating a lidar sensor (201) and a camera sensor (203) of a vehicle (200), comprising: determining (107) at least a first border (213) of at least one object, in the surroundings of the vehicle (200), that is delimited by the segmentation in the segmented lidar data (209); determining (109) at least a second border (217) of the object (215), in the surroundings of the vehicle (200), that is delimited by the segmentation in the segmented camera data (211); and calibrating (111) the lidar sensor (201) and the camera sensor by performing a border match optimisation operation between the first border (213), determined in the lidar data (205), and the second border (217), determined in the camera data (207), of the object (215).

Description

[0001] R. 416661

[0002] - 1 -

[0003] Description

[0004] title

[0005] Method for calibrating a lidar sensor and a camera sensor of a vehicle

[0006] The present invention relates to a method for calibrating a lidar sensor and a camera sensor of a vehicle.

[0007] State of the art

[0008] Methods for calibrating lidar sensors and camera sensors of vehicles are known from the state of the art.

[0009] It is an object of the present invention to provide an improved method for calibrating a lidar sensor and a camera sensor of a vehicle.

[0010] The problem is solved by the method for calibrating a lidar sensor and a camera sensor of a vehicle according to the independent claim. Advantageous embodiments are the subject of the dependent claims.

[0011] One aspect is provided: a computer-implemented method for calibrating a vehicle's lidar sensor and camera sensor, comprising:

[0012] Receiving lidar data from at least one lidar sensor of the vehicle, wherein the lidar data depict at least a part of the vehicle's surroundings; R. 416661

[0013] - 2 - Receiving camera data from at least one camera sensor of the vehicle, wherein the camera data depict at least the same part of the vehicle's surroundings as the lidar data;

[0014] Performing a segmentation of the lidar data and creating segmented lidar data;

[0015] Performing a segmentation of the camera data and creating segmented camera data;

[0016] Determining at least one first boundary of at least one object delimited by segmentation in the segmented lidar data in the vicinity of the vehicle;

[0017] Determining at least one second boundary of the object delimited by segmentation in the segmented camera data in the vicinity of the vehicle; and

[0018] Calibrating the lidar sensor and the camera sensor by performing a boundary matching optimization between the first boundary determined in the lidar data and the second boundary of the object determined in the camera data.

[0019] This provides the technical advantage of an improved method for calibrating a vehicle's lidar sensor and camera sensor. For this purpose, segmentations are first performed based on lidar data from the lidar sensor and camera data from the camera sensor, and objects positioned in the vehicle's vicinity are identified. Subsequently, based on the segmented lidar data, at least one boundary of at least one object is determined, whereby the first boundary is formed by data points of the lidar data within a boundary area of ​​the respective object. Similarly, a second boundary of the same object is determined based on the segmented camera data.

[0020] The calibration of the lidar sensor and the camera sensor is then performed taking into account the first and second boundaries by executing a boundary agreement optimization. By performing the boundary agreement optimization, a simple calibration of the lidar sensor and the camera sensor can thus be achieved. (R. 416661)

[0021] - 3 - are achieved in relation to each other. The boundary agreement optimization here means that the calibration of the lidar sensor and the camera sensor is carried out in such a way that the possibly differing first and second boundaries of the same object are brought into agreement.

[0022] According to the invention, the calibration of the lidar sensor and the camera sensor is designed as extrinsic calibration.

[0023] According to one embodiment, the first boundary of the object is defined as a plurality of data points of the lidar data that are arranged in a boundary area of ​​the object, and / or wherein the second boundary of the object is defined as a plurality of image pixels of the camera data that are arranged in the boundary area of ​​the object.

[0024] This offers the technical advantage that minimizing the distances between the first and second boundaries enables simple boundary alignment optimization and, consequently, simple and precise calibration of both sensors. The distances between the first and second boundaries are determined by the data points of the first boundary and the image pixels of the second boundary. This distance can be defined as a Euclidean distance, allowing for the simplest possible implementation of the distance minimization.

[0025] According to one embodiment, the boundary matching optimization comprises a minimization process of at least one distance between the first boundary of the lidar data and the second boundary of the camera data.

[0026] This achieves the technical advantage that the data points of the lidar data define a unique first boundary of the object within its perimeter area, and the image pixels of the camera data within its perimeter area provide a unique definition of the second boundary. R. 416661

[0027] - 4 -

[0028] According to one embodiment, determining the at least one second boundary in the camera data includes:

[0029] Generating a mask of the object delimited in the segmented camera data, wherein the mask is configured to cover at least one object in the camera data;

[0030] Generating a masking image based on the generated mask, wherein the masking image depicts the generated mask covering at least one object; and

[0031] Generating a border image based on the generated mask, where the second border is marked in the border image.

[0032] This offers the technical advantage that by generating a mask of the object delineated in the segmented camera data, a precise determination of the object's second boundary is possible. The boundary image is a graphical representation of the object's second boundary, determined based on the previously generated mask.

[0033] According to one embodiment, determining the at least one first boundary in the lidar data comprises:

[0034] Generating a mask of the object delimited in the segmented lidar data, wherein the mask is configured to cover data points in a central area of ​​the object and to leave the data points in the perimeter area of ​​the object uncovered; and

[0035] Applying the mask to the data points of the delimited object of the segmented lidar data and identifying the first boundary of the object as the data points of the boundary area of ​​the object that remain uncovered by the mask.

[0036] This allows for the technical advantage that, based on the mask of the object delineated in the segmented LiDAR data, a simple determination of the object's first boundary is possible. The mask is designed such that the data points in the central area of ​​the object are covered by the mask, while the data points in the R. 416661

[0037] - 5 - The boundary area is left untouched by the mask. By applying the mask to the lidar data, the first boundary can thus be generated in the form of the data points within the boundary area.

[0038] According to one embodiment, determining the at least one first boundary in the lidar data comprises:

[0039] Removing the data points from the segmented lidar data that do not belong to at least one object; and

[0040] Projecting the object-depicting data points of the segmented lidar data onto the camera data and generating an initial projection image; and / or including calibration:

[0041] Projecting the first outline onto the outline image and generating a second projection image.

[0042] This achieves the technical advantage that by removing data points from the segmented lidar data that do not belong to the selected object, the generation of the first outline is simplified by considering only the lidar data belonging to the respective object. Projecting the corresponding lidar data onto the camera data further simplifies the generation of the first outline. Projecting the generated first outline onto the outline image ensures that the first and second outlines are displayed together in the corresponding second projection image. This facilitates calibration by performing outline matching optimization.

[0043] According to one embodiment, in the border image, each pixel depicting the second border has a predefined pixel value, wherein pixels that do not depict the second border have pixel values ​​that deviate from the predefined pixel value, wherein the deviating pixel values ​​are dependent on the distance of the respective pixel to at least one pixel depicting the second border, wherein each data point of the lidar data projected into the border image is assigned a pixel value of the respective pixel of the border image onto which the respective data point was mapped, and wherein, for the implementation of the border matching R. 416661

[0044] - 6 - Optimization between the first border and the second border: Positions of the data points of the first border in the second projection image are varied in such a way that an alignment of the pixel values ​​assigned to the data points in the respective positions with the predefined pixel value of the pixels depicting the second border is achieved.

[0045] This allows for the technical advantage of simple border matching optimization. For this purpose, a predefined pixel value is assigned to each pixel in the border image that represents the second border.

[0046] Those pixels in the border image that do not represent the second border are assigned correspondingly different pixel values. The pixel values ​​of the individual pixels can be defined as brightness values. The predefined pixel value can be a maximum or minimum brightness value.

[0047] As the distance of the respective pixels to the pixels representing the second border increases, the pixel values ​​of those pixels that do not represent the second border show a higher deviation from the predefined pixel value of the pixels representing the second border.

[0048] To perform the border matching optimization of the first and second borders, each data point of the first border projected onto the border image is assigned the pixel value of the pixel onto which the respective data point of the first border was mapped.

[0049] The boundary matching optimization is now performed by varying the positions of the projected data points in such a way that the pixel values ​​assigned to the data points—that is, the pixel values ​​of those pixels onto which the data points were projected or mapped in the position variation—are aligned with the predefined pixel value of the pixels representing the second boundary. R. 416661

[0050] - 7 - By aligning the pixel values ​​assigned to the data points with the predefined pixel value of the pixels depicting the second border, the position of the data points within the second projection image can be approximated to the second border depicted in the second projection image.

[0051] The variations in the positions of the individual data points can be made taking into account the other data points of the first outline, so that an entire shift of the first outline projected into the second projection image takes place without changing the shape of the first outline.

[0052] According to one embodiment, the positions of the data points projected into the binary boundary image are varied taking into account gradient values ​​of the boundary image, and wherein the gradient values ​​describe changes in the pixel values ​​relative to the predefined pixel value depending on the distances of the respective pixels to the pixels depicting the second boundary.

[0053] This allows for the technical advantage that, taking into account the gradient values ​​of the boundary image, a precise variation of the positions of the individual data points of the first boundary within the second projection image is possible to align the pixel values ​​with the predefined pixel value.

[0054] By considering the respective gradient values, the direction in which the position change of a given data point of the first boundary must be performed in order to reach a pixel representing the second boundary can be determined. For this purpose, a gradient method known from the prior art can be used, for example.

[0055] According to one embodiment, the outline image is generated by applying a Canny-Edge algorithm to the generated mask in the camera data.

[0056] This allows the technical advantage to be achieved that, by applying the Canny-Edge algorithm, a border image with a precise representation can be generated. R. 416661

[0057] - 8 - of the second boundary can be generated. In particular, the gradient values ​​of the pixel values ​​can be generated by applying a distance transformation method to the boundary image. This allows distance information in the form of different pixel values ​​depending on the respective distance to the first boundary to be generated and integrated into the boundary image.

[0058] According to one embodiment, generating the outline image includes performing a high-gamma correction.

[0059] This offers the technical advantage of further improving the precision of the outline image by performing high-gamma correction. The gradient values ​​of the outline image can be generated, in particular, by performing the distance transformation method. This method assigns pixel values ​​to the individual pixels of the outline image based on their position relative to the initial outline.

[0060] According to one embodiment, performing the high-gamma correction includes applying a Gaussian blur filter.

[0061] This offers the technical advantage that applying a Gaussian blur filter allows for a clear representation of the second outline within the outline image. The application of the Gaussian blur filter creates a smooth gradient.

[0062] According to one embodiment, the method comprises:

[0063] Performing lidar odometry based on the lidar data from at least one lidar sensor;

[0064] Performing motion compensation of the lidar sensor during the acquisition of lidar data by the lidar sensor.

[0065] This allows the technical advantage to be achieved that by performing lidar odometry and motion compensation, the lidar data can be displayed on R. 416661

[0066] - 9 - Movement of the lidar sensor during lidar data acquisition can be corrected. This prevents motion distortion of the lidar data.

[0067] According to one embodiment, the segmentation of the lidar data and / or the segmentation of the camera data is performed by at least one trained artificial intelligence, in particular a segmentation network.

[0068] This allows for the technical advantage that, through appropriately trained artificial intelligence, precise segmentation of the lidar data and / or camera data is possible.

[0069] According to one aspect, a computing unit is provided which is set up to execute the procedure for calibrating a lidar sensor and a camera sensor of a vehicle according to one of the preceding embodiments.

[0070] According to one aspect, a computer program product is comprehensively provided with commands which, when the program is executed by a data processing unit, cause it to perform the procedure for calibrating a lidar sensor and a camera sensor of a vehicle according to one of the preceding embodiments.

[0071] Embodiments of the invention are described with reference to the following figures. The figures show:

[0072] Fig. 1 shows a schematic representation of a system for calibrating a lidar sensor and a camera sensor of a vehicle according to one embodiment;

[0073] Fig. 2 graphical representations of process steps of a method for calibrating a lidar sensor and a camera sensor of a vehicle according to one embodiment; R. 416661

[0074] - 10 - Fig. 3 further graphical representations of further process steps of a method for calibrating a lidar sensor and a camera sensor of a vehicle according to an embodiment;

[0075] Fig. 4 shows further graphical representations of further process steps of a method for calibrating a lidar sensor and a camera sensor of a vehicle according to one embodiment;

[0076] Fig. 5 shows further graphical representations of further process steps of a method for calibrating a lidar sensor and a

[0077] Fig. 6 shows a flowchart of the procedure for calibrating a lidar sensor and a camera sensor of a vehicle according to one embodiment; and

[0078] Fig. 7 is a schematic representation of a computer program product.

[0079] Fig. 1 shows a schematic representation of a system for calibrating a lidar sensor 201 and a camera sensor 203 of a vehicle 200 according to one embodiment.

[0080] Figure 1 shows a vehicle 200 on a road 237. The vehicle 200 comprises at least one lidar sensor 201 and one camera sensor 203 for monitoring the vehicle 200's surroundings. The vehicle 200 further comprises a computing unit 235 on which a calibration module 239 can be implemented, which is configured to execute the inventive method for calibrating the lidar sensor 201 and the camera sensor 203.

[0081] Alternatively, the method according to the invention can also be carried out on an external computing unit.

[0082] For this purpose, the calibration module 239 receives lidar data 205 from the lidar sensor 201 and camera data 207 from the camera sensor 203. R. 416661

[0083] - 11 - Lidar data 205 and camera data 207 each depict the environment of the vehicle 200.

[0084] Based on the lidar data 205 and the camera data 207, the calibration module 239 performs a segmentation and generates segmented lidar data 209 and segmented camera data 211. In the segmented lidar data 209 and the segmented camera data 211, the at least one object 215 displayed in the vicinity of the vehicle 200 is distinguished from the background of the environment as a corresponding object.

[0085] Based on the segmented lidar data 209, the calibration module 239 generates a first outline 213. The first outline 213 represents data points of the lidar data 205, which are arranged in an outline area 221 of the object 215, which is positioned in the vicinity of the vehicle 200 and imaged by the lidar data 205.

[0086] Based on the segmented camera data 211, the calibration module 239 generates a second outline 217 of the object 215.

[0087] According to one embodiment, the second boundary 217 is given by image pixels of the camera data 207, which represent the boundary area 221 of the object 215.

[0088] Based on the first and second outlines 213, 217 of the at least one object 215 arranged in the vicinity of the vehicle 200 thus determined, the calibration module 239 performs the calibration of the lidar sensor 201 and the camera sensor 203 by performing an outline matching optimization.

[0089] According to one embodiment, this can be achieved by minimizing the distance between the first border 213 and the second border 217.

[0090] Based on the appropriately executed boundary matching optimization, the Lidar sensor 201R. 416661

[0091] - 12 -and / or the camera sensor 203 are calibrated such that, after calibration of the lidar sensor 201 and / or camera sensor 203, the first and second outlines 213, 217 of the object 215 are matched as completely as possible.

[0092] Fig. 2 shows graphical representations of process steps of a method for calibrating a lidar sensor 201 and a camera sensor 203 of a vehicle 200 according to an embodiment.

[0093] In the graphs a) to c) different process steps for generating the first outline 213 based on the lidar data 205 are shown.

[0094] Figure a) shows the segmented lidar data 209. According to the invention, the lidar data 205 are formed as point clouds of individual data points of the lidar sensor 201.

[0095] Figure a) shows a section of the area surrounding vehicle 200, in which various objects 215, in this example other vehicles, are arranged. Figure a) shows the segmented point clouds of the data points 219 of the lidar data 209, which represent the various objects 215 in the area surrounding vehicle 200.

[0096] The point clouds shown for data points 219 exhibit central regions 231 and boundary regions 221. The central regions 231 represent areas of the depicted objects 215 within a central body area of ​​the respective objects 215. The boundary regions 221, on the other hand, are formed by the data points 219 that are arranged at the edge of the respective object 215 and represent the transition between different segmentations within the segmented lidar data 209.

[0097] According to the embodiment shown, the calibration module 239 generates a mask 229 for each object 215 of the segmented lidar data 209 based on the segmented lidar data 209. The respective mask 229 represents the respective object 215 and is configured, in particular, to cover the central area 231 of the respective object 215, the data points R. 416661

[0098] - 13 - 219 in the perimeter area 221 of the respective object 215, however, to be left uncovered.

[0099] The mask 229 of the data points 219 of the lidar data 209 can be generated by performing a dilation process followed by an erosion process. In the dilation process, each data point 219 is expanded in size such that the data points 219 overlap with each other, and gaps between the data points 219 of the lidar data 209 point cloud are closed. This generates a continuous image of the object represented by the lidar data 209 point cloud. In the subsequent erosion process, the continuous image of the represented object is reduced in size so that the size of the resulting mask 229 is smaller than the dimensions of the point cloud of data points 219 of the lidar data 209 representing the object 215. The shape of the mask 229 remains unchanged and continues to represent the object 215.

[0100] Based on the generated mask 229, the first boundary 213 is created by extracting the data points 219 of the central area 231 covered by the mask 229 from the point clouds of the data points 219 representing the respective objects 215, and considering only the data points of the boundary area 221. The correspondingly generated first boundary 213 is thus defined by the data points 219 of the boundary areas 221 of the respective objects 215.

[0101] According to the embodiment shown, in Figure a), the data points 219 of the segmented lidar data 209 that do not belong to the objects 215 selected for the calibration of the lidar sensor 201 and the camera sensor 203 have first been removed. Accordingly, Figure a) shows only the data points 219 that represent the respective objects 215.

[0102] According to a further embodiment, prior to the segmentation of the lidar data 205, lidar odometry is first performed based on the received lidar data 205, and motion compensation of the lidar sensor 201 is effected during the acquisition of the lidar data 205. This is R. 416661

[0103] - 14 -falsifications of the lidar data 205 based on the movement of the lidar sensor 201 due to the movement of the vehicle 200 are corrected.

[0104] In the embodiment shown in Figure a), to generate the first outline 213, the data points 219, which depict the respective objects 215, are first projected onto the camera data of the camera sensor 203. This creates a first projection image 233. The first projection image 233 is a combination of the camera data 207 and the data points 219 of the lidar data 205, which represent the respective objects and are projected onto it.

[0105] The objects 215 required for carrying out the calibration in the vicinity of the vehicle can be freely chosen.

[0106] Fig. 3 shows graphical representations of process steps of a method for calibrating a lidar sensor 201 and a camera sensor 203 of a vehicle 200 according to an embodiment.

[0107] The graphics a) to c) represent different steps for generating the second outline 217 based on the camera data 207.

[0108] Figure a) shows the segmented camera data 211, in which the objects 215 shown in the environment, in the example shown these are again the vehicles shown, are separated from the background of the environment by the segmentation through semantic segmentation.

[0109] Based on the segmented camera data 211, the calibration module 239 then generates a mask 223, with a separate mask 223 being generated for each detected object 215. The mask is configured such that each mask completely covers the object 215 of the segmented camera data 211.

[0110] Based on the mask 223 thus generated, a border image 227, shown in Figure c), is generated. The border image 227 shows only the second border 217 generated using mask 223. R. 416661

[0111] - 15 -

[0112] According to one embodiment, the outline image 227 is generated by performing a high-gamma correction.

[0113] According to one embodiment, the implementation of the high-gamma correction further includes the implementation of a Gaussian soft focus filter.

[0114] According to one embodiment, the outline image 227 is generated by applying a Canny-Edge algorithm to the generated mask 223 in the segmented camera data 211.

[0115] The outline image 227 exclusively depicts the second outlines 217 of the objects 215. No other features are shown in the outline image 227.

[0116] The depicted second borders 217 are represented exclusively in the form of different pixel values. Each pixel P1 that is part of the second border 217, and thus at least partially represents the second border 217, is assigned a predefined pixel value. This predefined pixel value can be defined as a maximum or minimum brightness value.

[0117] Those pixels P2 that do not depict the second outlines 217 shown, however, are assigned pixel values ​​that differ from the predefined pixel value.

[0118] In this case, the differing pixel values ​​of pixels P2, which do not represent the second border 217, show a greater deviation from the predefined pixel value with increasing distance to the pixels P2 that do represent the second border 217.

[0119] Pixels P1, which map the second outline 217, exhibit a minimum brightness value. The further the pixels P2 are from the position of the second outline 217, the greater the brightness value of the respective pixels becomes. R. 416661

[0120] - 16 -

[0121] According to one embodiment, the border image 227 may further include gradient information regarding the pixel values ​​of the individual pixels P1, P2 of the border image 227 in relation to the predefined pixel value that represents the second border 217 pixel P1.

[0122] In graphic c), this is illustrated by the gradual brightness gradient. The second outline, representing pixels P1 (217), is shown with minimum brightness, i.e., black. As the distance to the second outline 217 increases, the brightness of the respective pixels P2 gradually increases.

[0123] The outline image 227 thus represents a visual representation of the second outline 217 in the vicinity of the vehicle 200.

[0124] Fig. 4 shows graphical representations of process steps of a method for calibrating a lidar sensor 201 and a camera sensor 203 of a vehicle 200 according to an embodiment.

[0125] Figure 4 shows further process steps for calibrating the lidar sensor 201 and the camera sensor 203.

[0126] Figure a) shows a second projection image 243. In the second projection image 243, a projection of the data points 219 of the first boundary 213 onto the boundary image 227 is shown.

[0127] The second projection image 243 is generated by projecting the first outline 213 onto the outline image 225.

[0128] In the second projection image 243, both the first boundary 213, in the form of the data points 219 of the lidar data 201 in the boundary area 221 of the respective object 215, and the second boundary 217 in the form of the graphic representation within the boundary image 225 are shown.

[0129] Based on the first and second outlines 213, 217 shown in the second projection image 243, the following is subsequently done by R. 416661

[0130] - 17 - Calibration module 239 the calibration was carried out by performing the border matching optimization between the first and second borders 213, 217.

[0131] According to one embodiment, to perform the border matching optimization in the second projection image 243 between the first border 213 and the second border 217, the positions of the data points 219 of the first border 213 in the second projection image 243 are varied in such a way that the pixel values ​​assigned to the data points 219 of the first border 213 are aligned with the predefined pixel value of the pixels representing the second border 217.

[0132] As already described in Figure 3, the individual pixels P1 and P2 of the border image are assigned 227 brightness values. The pixels P1, which depict the second border 217, are assigned a maximum or minimum brightness value, i.e., a maximum or minimum pixel value.

[0133] In the illustrated embodiment, the pixels P1 representing the second border 217 are shown in black and thus have a minimum brightness value. With increasing distance from the second border 217, the pixels P2 become correspondingly brighter and thus have a higher brightness value, i.e., a higher pixel value.

[0134] After projecting the first outline 213 into the outline image 227 to generate the second projection image 243, each data point 219 of the first outline 213 is assigned a corresponding pixel value of the pixel P3 on which the respective data point 219 was mapped.

[0135] To perform the border matching optimization, the positions of data points 219 of the first border 213 within the second projection image 243 are varied such that the pixel values ​​assigned to the data points 219 in the varied positions are aligned with the predefined pixel value of the second border 217. R. 416661

[0136] - 18 - In the embodiment shown, the data points 219 are now varied in their position in such a way that the pixel values ​​assigned to the data points are minimized, i.e., the respective brightness values ​​are minimized.

[0137] The data points of the first boundary 213 can be varied in their position as a whole, so that the first boundary 213 as a whole object is shifted within the second projection image 243.

[0138] According to one embodiment, when varying the positions of the data points 219 of the first boundary 213, the gradient values ​​of the boundary image 227, i.e., of the second projection image 243, are taken into account.

[0139] The alignment of the pixel values ​​assigned to the data points with the predefined pixel value of the pixel P1 depicting the second boundary 217 can be achieved using a gradient method.

[0140] By taking into account the gradient values, which describe the deviation of the pixel values ​​from the predefined pixel value taking into account the respective distance of the pixels to each other, it is possible to vary the positions of the data points 219 in the direction of the pixels P1 that depict the second boundary 217.

[0141] By taking the pixel values ​​into account to perform the border matching optimization, the distance between the pixels P1 representing the second border 217 and the pixels P3 representing the first border is indirectly minimized.

[0142] For the distance minimization performed in this way, it is not necessary to explicitly calculate a distance between the respective pixels representing the first and second borders 213 and 217. This simplifies the optimization process.

[0143] According to one embodiment, minimizing the distance of R. 416661 can be used to optimize the boundary conformity.

[0144] - 19 - Distances A between the first and second borders 213, 217 are caused.

[0145] According to one embodiment, the distances to be minimized are defined as distances between the data points 219 of the first boundary 213 and the image pixels of the second projection image 233, which depict the second boundary 217.

[0146] According to one embodiment, the distance A is designed as a Euclidean distance.

[0147] Figure a) shows, in addition to the first outline 213, a corrected first outline 241. The corrected first outline represents the data points 219 of the first outline 213 after successful calibration of the lidar sensor 201 and the camera sensor 203.

[0148] As can be seen in Figure a), after calibration the agreement between the corrected first outline 241 and the second outline 217 is optimized, resulting in an almost complete agreement between the corrected first outline 241 and the second outline 217. This is equivalent to an optimized calibration of the lidar sensor 201 and the camera sensor 203.

[0149] Figure b) shows a further visualization of the successful calibration according to the procedure steps described above. In Figure b), the data points 219 of the first outline 213 and the corrected first outline 241 are shown on the camera data 207 of the camera sensor 203.

[0150] As is evident from Figure b), the calibration of the lidar sensor 201 and the camera sensor 203 based on the execution of the boundary matching optimization based on the first and second boundaries 213, 217 results in the corrected first boundary 241 representing a precise image of the outlines of the depicted objects 215, i.e., the depicted vehicles. R. 416661

[0151] - 20 - Fig. 5 shows graphical representations of process steps of a method for calibrating a lidar sensor 201 and a camera sensor 203 of a vehicle 200 according to an embodiment.

[0152] Figures a) and b) again show the successful calibration according to the procedure steps described above. Figure a) shows a projection of the data points 219 of the lidar data 205, which depict the objects 215, onto the camera data 207 before the calibration of the lidar sensor 201 and the camera sensor 203 was carried out.

[0153] In contrast, graphic b) shows the projection of the data points 219 of the lidar data 205 representing the objects 215 onto the camera data 207 after performing the calibration by executing the boundary matching optimization according to the steps described above. As can be clearly seen in graphic b), the data points 219 of the lidar data 205 represent a much more precise image of the respective objects 215 after the calibration has been performed.

[0154] Fig. 6 shows a flowchart of the method 100 for calibrating a lidar sensor 201 and a camera sensor 203 of a vehicle 200 according to an embodiment.

[0155] To calibrate the lidar sensor 201 and the camera sensor 203 of the vehicle 200, in a first procedure step 101 the lidar data 205 of the at least one lidar sensor 201 are first received.

[0156] In process step 129, lidar odometry is performed based on the lidar data 205.

[0157] In a further process step 131, a motion compensation of the lidar data 205 of the lidar sensor 201 is carried out with respect to a movement of the lidar sensor 201 during the recording of the lidar data 205.

[0158] In process step 103, the camera data 207 from the camera sensor 203 are received. R. 416661

[0159] - 21 -

[0160] In process step 105, a segmentation of the lidar data 205 and a generation of segmented lidar data 209 are carried out.

[0161] In process step 107, a segmentation of the camera data 207 and a generation of segmented camera data 211 are performed.

[0162] In a process step 109, the first boundary 213 is determined based on the segmented lidar data 209.

[0163] In a process step 123, the data points 219 are first removed from the segmented lidar data 209 that do not represent the objects 215 selected for calibration in the vicinity of the vehicle 200.

[0164] In a further process step 125, the data points 219 representing the objects 215 of the segmented lidar data 209 are projected onto the camera data 205 and a first projection image 233 is generated.

[0165] In a further process step 119, the mask 229 is generated based on the data points 219 representing the objects 215.

[0166] In a further process step 121, the mask 229 is applied to the data points 219 of the respective object 215 and the first outline 213 is identified as the data points 219 in the outline area 221 of the respective object 215 as the first outline 213.

[0167] In a further process step 111, the second boundary 217 is determined based on the segmented camera data 211.

[0168] In a further process step 115, a corresponding mask 223 is created for each object 215.

[0169] In a further process step 117, a masking image 225 is generated based on the created mask 223. The masking image 125 represents at least the mask 223 covering the object 215. R. 416661

[0170] - 22 -

[0171] In a further process step 113, the calibration of the lidar sensor 201 and the camera sensor 203 is calibrated based on the first and second outlines 213, 217 by performing an outline matching optimization.

[0172] In a process step 127, the first outline 213 is projected onto the outline image 227 and a second projection image 243 is generated.

[0173] Based on the second projection image 243, the boundary matching optimization is then carried out.

[0174] For this purpose, the distance minimization between the distances A between the data points 219 of the first boundary 213 and the image pixels of the second projection image 243 representing the second boundary 217 can be determined.

[0175] Fig. 7 shows a schematic representation of a computer program product 300, comprising instructions which, when the program is executed by a data processing unit, cause it to execute the method 100 for calibrating the lidar sensor 201 and the camera sensor 203.

[0176] In the embodiment shown, the computer program product 300 is stored on a storage medium 301. The storage medium 301 can be any storage medium known from the prior art.

Claims

R. 416661 - 23 - Claims 1. Computer-implemented method (100) for calibrating a lidar sensor (201) and a camera sensor (203) of a vehicle (200), comprising: Receiving (101) lidar data (205) from at least one lidar sensor (201) of the vehicle (200), wherein the lidar data (205) represent at least a part of the vehicle's (200's) surroundings; Receiving (103) camera data (207) from at least one camera sensor of the vehicle (200), wherein the camera data (207) represent at least the same part of the vehicle's (200) surroundings as the lidar data (205); Performing (105) a segmentation of the lidar data (205) and creating segmented lidar data (209); Performing (107) a segmentation of the camera data (207) and creating segmented camera data (211); Determine (109) at least one first boundary (213) of at least one object delimited by segmentation in the segmented lidar data (209) in the vicinity of the vehicle (200); determine (111) at least one second boundary (217) of the object (215) delimited by segmentation in the segmented camera data (211) in the vicinity of the vehicle (200); and calibrate (113) the lidar sensor (201) and the camera sensor by performing boundary matching optimization between the first boundary (213) determined in the lidar data (205) and the second boundary (217) of the object (215) determined in the camera data (207).

2. Method (100) according to claim 1, wherein the first boundary (213) of the object (215) is defined as a plurality of data points (219) of the lidar data (205) arranged in a boundary area (221) of the object (215), and / or wherein the second boundary (217) of the object (215) is defined as a plurality of image pixels of the camera data (207) arranged in the boundary area (221) of the vehicle (200). R. 416661 - 24 - 3. Method (100) according to claim 1 or 2, wherein the boundary matching optimization comprises a minimization process of at least one distance (A) between the first boundary (213) of the lidar data (205) and the second boundary (217) of the camera data (207).

4. Method (100) according to one of the preceding claims, wherein the determination (111) of the at least one second boundary (217) in the camera data (207) comprises: Generating (115) a mask (223) of the object (215) delimited in the segmented camera data (211), wherein the mask (223) is configured to cover the at least one object (215) in the camera data (207); generating (117) a masking image (225) based on the generated mask (223), wherein the masking image (225) represents the mask (223) covering the at least one object (215); and Generating (119) a border image (227) based on the generated mask (223), wherein the second border (217) is indicated in the border image (227).

5. Method (100) according to one of the preceding claims, wherein the determination (109) of the at least one first boundary (213) in the lidar data (205) comprises: Generating (121) a mask (229) of the object (215) delimited in the segmented lidar data (209), wherein the mask (229) is configured to cover data points (219) in a central area (231) of the object (215) and to leave the data points (219) in the boundary area (221) of the object (215) uncovered; and Applying (123) the mask (229) to the data points (219) of the delimited object (215) of the segmented lidar data (209) and identifying the first boundary (213) of the object (215) as the data points (219) of the boundary area (221) of the object (215) that remain uncovered by the mask (229). R. 416661 - 25 - 6. Method (100) according to claim 5, wherein the determination (109) of the at least one first boundary (213) in the lidar data (205) comprises: Removing (125) the data points (219) of the segmented lidar data (209) that do not belong to the at least one object (215); and projecting (127) the data points (219) of the segmented lidar data (209) representing the object (215) onto the camera data (205) and generating a first projection image (233); and / or wherein the calibration (113) comprises: Projecting (129) the first outline (213) onto the outline image (227) and generating a second projection image (243).

7. Method (100) according to claim 6, wherein in the border image (227) each pixel (P1) depicting the second border (217) has a predefined pixel value, wherein pixels (P2) that do not depict the second border (217) have pixel values ​​that deviate from the predefined pixel value, wherein the deviating pixel values ​​are dependent on the distance of the respective pixel (P2) to at least one pixel (P1) depicting the second border (217), wherein each data point (219) of the lidar data (209) projected onto the border image (227) is assigned a pixel value of the respective pixel (P3) of the border image (227) onto which the respective data point (219) was mapped, and wherein, for the purpose of performing border matching optimization between the first border (213) and the second border (217), positions of the data points (219) the first outline (213) in the second projection image (243) are varied in such a way,that an alignment of the pixel values ​​assigned to the data points (219) in the respective positions with the predefined pixel value of the pixels (P19) depicting the second border (217) is effected.

8. Method (100) according to claim 7, wherein the positions of the data points (219) projected onto the border image (227) are varied taking into account gradient values ​​of the border image (227), and wherein the gradient values ​​are changes in the pixel values ​​relative to the predefined pixel value depending on the distances of the R. 416661 - 26 - describe the respective pixels (P2) to the pixels (P1) that depict the second border (217).

9. Method (100) according to any one of the preceding claims 4 to 8, wherein the outline image (227) is generated by applying a Canny-Edge algorithm to the generated mask (223) in the camera data (207).

10. Method (100) according to any one of the preceding claims 4 to 9, wherein generating the border image (227) comprises performing a high-gamma correction.

11. Method (100) according to claim 10, wherein performing the high-gamma correction comprises performing a Gaussian blur filter.

12. Method (100) according to any one of the preceding claims, further comprising: Performing (131) lidar odometry based on the lidar data (205) of the at least one lidar sensor (201); Performing (133) motion compensation of the lidar sensor (201) during the acquisition of lidar data (205) by the lidar sensor (201).

13. Method (100) according to one of the preceding claims, wherein the segmentation of the lidar data (205) and / or the segmentation of the camera data (207) is carried out by at least one trained artificial intelligence, in particular a segmentation network.

14. Computing unit (235) configured to perform the method (100) for calibrating a lidar sensor (201) and a camera sensor (203) of a vehicle (200) according to any one of the preceding claims 1 to 13.

15. Computer program product comprising instructions which, when the program is executed by a data processing unit, cause it to perform certain actions, R. 416661 - 27 - the method (100) for calibrating a lidar sensor (201) and a camera sensor (203) of a vehicle (200) according to any one of the preceding claims 1 to 13.

Images