Method and devices for calibrating a motor vehicle
The use of image-based segmentation algorithms for identifying geometric objects in vehicles enables accurate and efficient calibration of driver assistance systems, overcoming the limitations of traditional target-based methods by allowing automated calibration in diverse environments.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-12-12
- Publication Date
- 2026-06-25
AI Technical Summary
Existing calibration methods for driver assistance systems in vehicles are complex, require specific environmental conditions, and are prone to errors due to incorrect target placement, limiting their applicability and accuracy.
A method using image-based segmentation algorithms to identify geometric objects in 2D images and 3D point clouds, allowing for automated calibration without the need for predefined targets, enhancing flexibility and robustness against environmental variations.
The method provides accurate and efficient calibration by eliminating the need for target placement, reducing human error, and enabling calibration in various environments with improved precision and reduced time and cost.
Smart Images
Figure EP2025086918_25062026_PF_FP_ABST
Abstract
Description
[0001] R.417430IP1
[0002] - 1 -
[0003] Description
[0004] title
[0005] Methods and devices for calibrating a motor vehicle
[0006] The invention relates to a method and a device for calibrating a driver assistance system of a motor vehicle with the features of the preamble of the independent claims.
[0007] State of the art
[0008] To calibrate the sensors of driver assistance systems, which are frequently installed in motor vehicles, calibration devices with one or more measuring targets ("calibration targets") are often used, particularly in workshops. Each calibration target can display at least one optical pattern that can be optically detected by at least one optical sensor of the driver assistance system being calibrated, in order to calibrate the driver assistance system or at least one sensor of the driver assistance system.
[0009] Besides target-based methods, there are also approaches based on edge detection or other environmental features. However, these methods are often complex to implement and limited in their applicability, as they require specific environmental characteristics and do not function reliably in arbitrary environments. The accuracy of the calibration depends heavily on the precise placement and detection of the targets and the quality of the extracted environmental features.
[0010] To perform the calibration correctly, the calibration device must be positioned in front of the vehicle at a location specified by the manufacturer of the driver assistance system. Document DE102023203531 A1 discloses a method for the automated positioning of a calibration device for driver assistance systems. Instead of manual R.417430IP1
[0011] - 2 -
[0012] For measurements and targets, the system uses an image capture device (e.g., a stereo camera) to capture characteristic features of the vehicle and extract 3D information. From this, the vehicle's longitudinal axis and distance to the device are determined in order to calculate and display the optimal position of the calibration device.
[0013] Disclosure of the invention
[0014] An inventive method for calibrating a motor vehicle comprises optically capturing an area of the motor vehicle to be evaluated by means of at least one image acquisition device and calculating a 3D point cloud of the area to be evaluated. The following steps are performed:
[0015] - Identifying at least one geometric object, in particular a surface, a curve or a point, in at least one 2D image of the area to be evaluated using an image-based segmentation algorithm; - Determining the position of the at least one geometric object in the 3D reference system of the at least one image acquisition device by projecting the geometric object identified in the at least one 2D image into the 3D point cloud;
[0016] -Calibrating the motor vehicle, whereby the position of the motor vehicle relative to the image recording device is determined using the position of at least one geometric object in the 3D reference system of the image recording device.
[0017] The method according to the invention enables calibration without special preparation, such as the application of targets. In particular, the use of an image-based segmentation algorithm increases flexibility and robustness, since the image-based segmentation algorithm can be trained on various geometric objects and is therefore more flexible than methods based on predefined targets or edges. The method according to the invention is thus also more robust against variations in lighting, textures, and minor damage or soiling of the vehicle.
[0018] Unlike target-based methods, there is no need to place and measure special reference objects. This saves time and costs and simplifies the calibration process. The risk of errors due to incorrect target placement is also eliminated. R.417430IP1
[0019] - 3 -
[0020] The entire process of object identification and location determination can be automated, making the calibration process more efficient and reproducible, and reducing human error.
[0021] By combining information from the 2D images with the data from the 3D point cloud, the position of the geometric object in the reference system of the at least one image acquisition unit can be precisely determined, leading to more accurate calibration. Furthermore, the position of the geometric object in the reference system of the calibration device can be determined if the position of the calibration device in the reference system of the at least one image acquisition unit is known. This can be achieved through a fixed connection between the at least one image acquisition unit and the calibration device, or by optically determining the position of the calibration device. For this purpose, the calibration device can, for example, incorporate optical targets.
[0022] The image-based segmentation algorithm can recognize specific features of different vehicle types in previously unknown images, making the method universally applicable.
[0023] Since no targets are required, calibration is possible in various environments without any special environmental requirements. Furthermore, the space requirement is minimal, allowing calibration even in small rooms or open areas.
[0024] The invention also includes a device for calibrating a motor vehicle. The device comprises at least one image acquisition device and one evaluation device. The image acquisition device is configured to optically capture an area of the motor vehicle to be evaluated and to calculate a 3D point cloud of the area to be evaluated. The evaluation device is configured to identify at least one geometric object, in particular a surface, a curve, or a point, in at least one 2D image of the area to be evaluated using an image-based segmentation algorithm; the position of the at least one geometric object in the 3D reference system of the at least one image acquisition device is determined by projecting the point cloud in the at least one 2D image.
[0025] - 4 - to determine the identified geometric object in the 3D point cloud; and to calibrate the motor vehicle, whereby the position of the motor vehicle relative to the image recording device is determined using the position of the at least one geometric object in the 3D reference system of the image recording device.
[0026] The device according to the invention has the same advantages as the method according to the invention.
[0027] The dependent claims describe advantageous embodiments and further developments of the method and device according to the invention.
[0028] The use of a deep learning method for the image-based segmentation algorithm is advantageous, stemming from its ability to autonomously learn to recognize complex patterns and features in image data. Unlike classical segmentation methods based on manually executed rules, a deep learning model, when trained on large datasets, can automatically extract the relevant features for identifying geometric objects.
[0029] The advantage of using a "Segment Anything Model" (SAM) as an image-based segmentation algorithm lies in its remarkable ability to segment any objects in an image.
[0030] It is advantageous if the position of a calibration device in front of the vehicle is determined based on the position of at least one geometric object, since the position of the geometric object can be chosen to represent important reference points of the vehicle. Thus, for example, the position of a front camera, representing the geometric object, or the position of a vehicle center plane, which can be determined from the geometric object, can be determined relative to the calibration device. Based on the position of the geometric object, in particular the vehicle center plane and / or the front camera, the size and / or orientation of an optical pattern / target displayed on a calibration target, in particular an electronic display device, can then be adjusted. This is automated and possible without user interaction, so that the calibration procedure can be simpler, faster, and less prone to errors.417430IP1.
[0031] - 5 -
[0032] It is advantageous if the area to be evaluated includes a windshield, a left A-pillar, and a right A-pillar, with the windshield surface segmented as the geometric object in the 2D image. From the 3D point cloud of the area to be evaluated and the windshield surface in the 2D image, the position of the left and right A-pillars is determined by a left curve and a right curve, respectively, in the 3D reference frame of the image acquisition device. A-pillars are well-defined, stable structures present in most motor vehicles and are therefore ideally suited as reference points for calibration. Furthermore, the windshield, as a homogeneous surface, can be very effectively segmented in a 2D image.
[0033] Determining the vehicle's center plane from the position of the left and right curves provides precise information about the vehicle's orientation in space. Using both A-pillars compensates for minor asymmetries in the vehicle's structure, resulting in a more robust determination of the vehicle's center plane.
[0034] It is advantageous if the at least one image acquisition device is attached to a calibration device with a calibration target, in particular with an electronic display device. Attaching the image acquisition device to the calibration device creates a compact and mobile unit that is easy to handle and transport. This reduces the space required and simplifies the calibration process. Furthermore, a fixed reference exists between the calibration target and the at least one image acquisition unit, so that the position of the geometric object, in particular the vehicle's center plane or the front camera, can be transferred from the reference system of the at least one image acquisition device to the reference system of the calibration target or the calibration device.
[0035] If there is no fixed relationship between the calibration target and the at least one image acquisition unit, this can advantageously be established by arranging the at least one image acquisition device such that the calibration device is within the field of view of the at least one image acquisition device. The calibration device can optionally be R.417430IP1
[0036] - 6 -
[0037] have markings to establish a spatial relationship between the at least one image acquisition device and the calibration device.
[0038] An embodiment of the invention is described below with reference to the accompanying drawings.
[0039] Brief description of the characters
[0040] Figure 1 shows a schematic top view of a measuring station with a motor vehicle and a calibration device;
[0041] Figure 2 shows a schematic representation of an area to be evaluated and a windshield identified using an image-based segmentation algorithm;
[0042] Figure 3 schematically shows the determination of a vehicle center plane from a 3D point cloud and a segmented 2D image;
[0043] Figure 4 shows a flow diagram of a method according to the invention in an exemplary embodiment.
[0044] Character description
[0045] Figure 1 shows a schematic top view of a measuring station 2 with a motor vehicle 4 equipped with a driver assistance system 16. The driver assistance system 16 has at least one sensor 18. The sensor 18 can, for example, be a front camera.
[0046] A calibration device 20 is positioned in front of the motor vehicle 4.
[0047] The calibration device 20 shown in Figure 1 comprises a central calibration target 12. An optical pattern / target, not visible in Figure 1, is formed on the calibration target 12. This pattern / target can be optically detected by the sensor 18 of the driver assistance system 16 in order to calibrate the sensor 18 and / or the driver assistance system 16. In the case of a radar sensor, the calibration target 12 is designed as a radar-reflecting calibration target 12. R.417430IP1
[0048] - 7 -
[0049] The calibration target 12 can also be configured as an electronic display device, wherein the electronic display device is configured to electronically display an optical pattern / target. Depending on the orientation of the motor vehicle 4 to the calibration target 12, the optical pattern / target can be enlarged, reduced, shifted, or distorted on the electronic display device in order to achieve optimal alignment of the optical pattern / target with respect to the sensor 18, in particular the front camera.
[0050] The device 7 according to the invention for calibrating a motor vehicle 4 comprises at least one image acquisition device 6a, 6b, which is designed to optically capture the motor vehicle 4 and to calculate a 3D point cloud of an area 31 to be evaluated.
[0051] The at least one image recording device 6a, 6b can, for example, comprise a stereo camera system 9 with a first image recording device 6a and a second image recording device 6b, which makes it possible to record three-dimensional images, in particular a 3D point cloud, of the motor vehicle 4.
[0052] The first image acquisition device 6a and the second image acquisition device 6b can each be attached to the right and left of the calibration plate 12 and be firmly connected to it.
[0053] In an alternative embodiment, the stereo camera system 9 can be positioned next to the calibration device 20, so that the stereo camera system 9 has both the calibration device 20 and the motor vehicle 4 in its field of view.
[0054] Other embodiments are also possible in which a single camera is moved and a 3D point cloud of the motor vehicle 4 is generated using the structure-from-motion method.
[0055] The device 7 according to the invention further comprises an evaluation device 8, which can perform the steps of the method according to the invention described below. R.417430IP1
[0056] - 8 -
[0057] According to the method according to the invention, the at least one image recording device 6a, 6b is arranged in front of the motor vehicle 4 in such a way that an area 31 of the motor vehicle 4 to be evaluated can be optically detected and a 3D point cloud can be calculated.
[0058] In addition to the 3D point cloud, a 2D image of the area to be evaluated 31 is also recorded and evaluated with the evaluation device 8.
[0059] In the 2D image, at least one geometric object 32, in particular a surface, a curve, or a point, is identified using an image-based segmentation algorithm. The image-based segmentation algorithm analyzes the 2D image and uniquely classifies each pixel, or, if a coarser resolution is used, a subset of the pixels, within the area 31 to be evaluated. Each pixel is assigned either membership in the segmented object, which corresponds to the geometric object 32, or membership in the background, thus creating a binary segmentation mask. This binary segmentation mask defines a precise boundary between the geometric object 32 and the rest of the 2D image.
[0060] The image-based segmentation algorithm is based on a deep learning method. For training, a dataset of 2D images was used, depicting different vehicles from various perspectives, under different lighting conditions, and with varying backgrounds. For each 2D image, a precise segmentation mask of the geometric object, such as the windshield, was created. This mask separates the pixels of the geometric object, such as the windshield, from other parts of the vehicle and the background.
[0061] A pre-trained deep learning algorithm was then fine-tuned using a specialized dataset of geometric objects, such as windshields. During this process, the model's weights were adjusted to maximize recognition accuracy. The inputs during training consisted of the images and their corresponding segmentation masks. These segmentation masks must be defined by the user.
[0062] According to one embodiment of the invention, the image-based segmentation algorithm is based on a SAM (Segment Anything Model) R.417430IP1
[0063] - 9 -
[0064] Algorithm. The SAM (Segment Anything Model) algorithm is an image segmentation model developed by Meta AI Research that aims to accurately segment any object in a 2D image, even if that object was unknown to the model during training.
[0065] After the position of the at least one geometric object 32, in particular the surface, the curve or the point, in the 2D image has been identified, the position of the at least one geometric object 32 in the 3D reference system of the at least one image acquisition device 6a, 6b is determined by projecting the geometric object 32 identified in the at least one 2D image into the 3D point cloud.
[0066] This is done by assigning the respective pixels in the 2D image that belong to the geometric object to the corresponding pixels in the 3D point cloud. In this way, the geometric object 32 identified in the 2D image can also be identified in the 3D point cloud.
[0067] The position of the at least one geometric object 32 in the reference system of the at least one image acquisition unit 6a, 6b can be used to calibrate the motor vehicle 4, since the orientation of the motor vehicle 4 to the image acquisition unit 6a, 6b can be determined with the help of the position of the at least one geometric object 32 in the 3D reference system of the image acquisition device 6a, 6b.
[0068] Figure 2 shows an embodiment of the method according to the invention. Here, a windshield 41, a left A-pillar 42, and a right A-pillar 43 of a motor vehicle 4 are recorded in a 2D image as the area 31 to be evaluated.
[0069] After the 2D image of the area 31 to be evaluated was transferred to the evaluation device 8, the windshield 41 was identified as geometric object 32 by an image-based segmentation algorithm. The image-based segmentation algorithm was trained to identify the area of a windshield 41. A result of the evaluation by a segment-anything algorithm is shown in the right-hand image. The area of the windshield 41 in the 2D image has been segmented as geometric object 32, while all other pixels 40 in the 2D image were ignored; these pixels are usually displayed as black. R.417430IP1
[0070] - 10 -
[0071] The area 31 to be evaluated, in particular the windshield 41, the left A-pillar 42 and the right A-pillar 43, was also captured by the at least one image recording device 6a, 6b and a 3D point cloud of the area 31 to be evaluated was calculated.
[0072] Figure 3 shows a 3D point cloud of the windshield 41, the left A-pillar 42, and the right A-pillar 43 in a 3D reconstruction image 53. In this 3D reconstruction image 53, each pixel is assigned a gray value that contains distance information. Furthermore, the surface of the windshield 41 has been segmented in the 2D image shown below the 3D reconstruction image 53 with the reference symbol 52. By projecting or overlapping the image information from the 3D reconstruction image 53 and the segmented 2D image 52, it is possible to determine the position of the left A-pillar 62 and the right A-pillar.
[0073] 63 each by a left curve and a right curve in the 3 D reference system 64 of the at least one image acquisition device 6a, 6b to be determined.
[0074] To calculate the left curve 62 and the right curve 63, the perimeter of the segmented area of the windshield 41 can be transferred from the 2D image to the 3D point cloud. In this process, the corresponding pixels of the 2D image are assigned to the pixels of the 3D point cloud. Based on the radius of curvature of the windshield 41's perimeter, the four corners of the windshield 41 can be identified, allowing the area of the segmented surface, particularly the perimeter, to be identified on the left (corresponding to the left curve 62) and right (corresponding to the right curve 63) of the windshield 41, which can then be assigned to the left A-pillar 42 and the right A-pillar 43, respectively.
[0075] From the position of the left curve 62 and the right curve 63 in the 3 D reference system 64 of the at least one image recording device 6a, 6b, the position of the vehicle median plane 65 can be determined, since this represents the plane of symmetry of the two curves 62,63 to each other.
[0076] The vehicle center plane 65 can be used to calibrate the motor vehicle 4, since the position of the vehicle center plane 65 in the 3D reference system can be used.
[0077] 64 of the image recording device 6a, 6b determines the orientation of the motor vehicle 4 to the image recording device 6a, 6b. R.417430IP1
[0078] - 11 -
[0079] In a further procedural step, the position of the motor vehicle 6, in particular the vehicle median plane 65, can also be determined relative to the calibration device 20, since the position of the calibration device 20 and the position of the vehicle median plane 65 in the coordinate system of the at least one image recording device 6a, 6b is known.
[0080] In a further alternative process step, the size and / or orientation of the geometric object / target on the calibration target 12, in particular the electronic display device, can also be adjusted based on the position of the at least one geometric object, in particular the vehicle center plane 65.
[0081] The evaluation device 8 can be configured to receive 2D images and / or 3D point clouds from the at least one image acquisition device 6a, 6b and to evaluate the 2D images and / or 3D point clouds provided by the at least one image acquisition device 6a, 6b according to the method according to the invention. The 3D point cloud can be calculated directly in the at least one image acquisition device 6a, 6b. In an alternative embodiment, the evaluation device can calculate the 3D point cloud from 2D images. The result can be displayed on a display device 10, which may, for example, include a screen and / or a printer.
[0082] The 2D images and / or 3D point clouds can be transmitted wirelessly, e.g. via a WLAN or BluetoothO data connection, or via a wired connection from the at least one image acquisition device 6a, 6b to the evaluation device 8.
[0083] At least one image acquisition device 6a, 6b, which is not attached to the calibration device 20, can be arranged on or at the measuring station 2 in such a way that the calibration device 20 is in the field of view of the at least one image acquisition device 6a, 6b.
[0084] Figure 4 shows a flowchart of a method according to the invention for calibrating a motor vehicle 4, comprising the following steps. R.417430IP1
[0085] - 12 -
[0086] In a process step 100, an area 31 of the motor vehicle 4 to be evaluated is optically recorded by at least one image recording device 6a, 6b.
[0087] In process step 110, a 3D point cloud of the area 31 to be evaluated is calculated.
[0088] In a process step 120, at least one geometric object 32, in particular a surface, a curve or a point, is identified in at least one 2D image of the area 31 to be evaluated using an image-based segmentation algorithm.
[0089] Procedure steps 110 and 120 can also be carried out in a different order or simultaneously.
[0090] In a process step 130, the position of the at least one geometric object 32 in the 3D reference system of the at least one image acquisition device 6a, 6b is determined by projecting the geometric object 32 identified in the at least one 2D image into the 3D point cloud.
[0091] In process step 140, the motor vehicle 4 is calibrated, taking into account the position of the at least one geometric object 32 in the 3D reference system of the image recording device 6a, 6b.
Claims
R.417430 - 13 - Patent claims 1. Method for calibrating a motor vehicle (4), wherein an area (31) of the motor vehicle (4) to be evaluated is optically detected by at least one image recording device (6a, 6b) and an SD point cloud of the area (31) to be evaluated is calculated, characterized in that the following steps are carried out: - Identifying at least one geometric object (32), in particular a surface, a curve or a point, in at least one 2 D image of the area to be evaluated (31) using an image-based segmentation algorithm; - Determining the position of the at least one geometric object (32) in the 3D reference system of the at least one image acquisition device (6a, 6b) by projecting the geometric object (32) identified in the at least one 2D image into the 3D point cloud; - Calibrating the motor vehicle (4), taking into account the position of the at least one geometric object (32) in the 3D reference system of the image acquisition device (6a, 6b).
2. The method according to claim 1, wherein the image-based segmentation algorithm is based on a deep learning method.
3. The method of claim 2, wherein the image-based segmentation algorithm is based on a Segment Anything Model.
4. Method according to one of the preceding claims, wherein a position of a calibration device (20) in front of the motor vehicle (4) is determined based on the position of the at least one geometric object.
5. Method according to one of the preceding claims, wherein a size and / or an orientation of an optical element displayed on a calibration target (12), in particular an electronic display device, is determined. R.417430 - 14 - Patterns / targets are adapted based on the position of at least one geometric object (32).
6. Method according to one of the preceding claims, wherein the area to be evaluated comprises a windshield (41) and a left A-pillar (42) and a right A-pillar (43), wherein a surface of the windshield is segmented in the 2D image as the geometric object (32), and the position of the left A-pillar (42) and the right A-pillar (43) is determined from the 3D point cloud of the area to be evaluated (31) and the surface of the windshield (41) in the 2D image by a left curve (52) and a right curve (53) in the 3D reference system of the image acquisition device (6a, 6b).
7. Method according to claim 6, wherein a vehicle median plane is determined from the position of the left curve (52) and the right curve (53).
8. Device (7) for calibrating a motor vehicle (4), wherein the device (7) comprises: at least one image acquisition device (6a, 6b) configured to optically capture an area of the motor vehicle to be evaluated and to calculate a 3D point cloud of the area to be evaluated or a section thereof; an evaluation device (8) configured to identify at least one geometric object, in particular a surface, a curve or a point, in at least one 2D image of the area to be evaluated (31) using an image-based segmentation algorithm; and to determine the position of the at least one geometric object (32) in the 3D reference system of the at least one image acquisition device (6a, 6b) by projecting the at least one geometric object (32) identified in the 2D image onto the 3D point cloud;and which is further equipped to determine the orientation of the motor vehicle (4) to the image recording device (6a, 6b) and to calibrate the motor vehicle (4) using the position of the at least one geometric object (32) in the 3D reference system of the image recording device (6a, 6b). R.417430 - 15 - 9. Device (7) according to claim 8, wherein the at least one image acquisition device (6a, 6b) is attached to a calibration device (20) with a calibration target (12), in particular with an electronic display device.
10. Device (7) according to claim 8, wherein the at least one image acquisition device (6a, 6b) is arranged such that the calibration device (20) is in the field of view of the at least one image acquisition device (6a, 6b).