Method and devices for determining the position of a front camera of a motor vehicle
An image-based segmentation algorithm for vehicle sensors addresses the complexity and error-prone nature of existing calibration methods by accurately determining the front camera's position, enabling efficient and flexible calibration across diverse environments.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-12-08
- Publication Date
- 2026-06-25
AI Technical Summary
Existing calibration methods for vehicle sensors, such as LiDAR and radar, are complex, environment-dependent, and prone to errors due to precise target placement and environmental feature detection, limiting their applicability and accuracy.
An image-based segmentation algorithm, trained on geometric objects, is used to identify characteristic surfaces of a vehicle's front camera, generating a 3D point cloud to determine its position accurately, eliminating the need for precise target placement and enhancing flexibility and robustness.
The method provides accurate, efficient, and universally applicable calibration by automating the process, reducing human error, and allowing calibration in various environments without specific environmental requirements, thus improving sensor calibration accuracy and efficiency.
Smart Images
Figure EP2025085864_25062026_PF_FP_ABST
Abstract
Description
[0001] R.417431
[0002] - 1 -
[0003] Description
[0004] title
[0005] Methods and devices for determining the position of a front camera of a motor vehicle
[0006] The invention relates to a method and a device for determining the position of a front camera of a motor vehicle with the features of the preamble of the independent claims.
[0007] State of the art
[0008] To calibrate the sensors of driver assistance systems, which are frequently installed in motor vehicles, calibration devices with one or more measuring targets ("calibration targets") are often used, particularly in workshops. Each calibration target can display at least one optical pattern that can be optically detected by at least one optical sensor of the driver assistance system being calibrated, in order to calibrate the driver assistance system or at least one sensor of the driver assistance system.
[0009] For the calibration of vehicle cameras and other vehicle sensors, such as LiDAR or radar, reference objects, so-called targets, with known dimensions and positions are frequently used. The distance between the vehicle and the calibration plane is determined by image-based detection of these targets. Besides target-based methods, approaches also exist that are based on edge detection or other features in the environment. However, these methods are often complex to implement and limited in their applicability, as they require specific environmental features and do not function reliably in all environments. The accuracy of the calibration depends heavily on the precise placement and detection of the targets and / or the quality of the extracted environmental features. R.417431
[0010] - 2 -
[0011] To perform the calibration correctly, the calibration device must be positioned in front of the vehicle at a location specified by the manufacturer of the driver assistance system. Document DE102023203531 A1 discloses a method for the automated positioning of a calibration device for driver assistance systems. Instead of manual measurements and targets, the system uses an image acquisition device (e.g., a stereo camera) to capture characteristic features of the vehicle and extract 3D information. From this, the vehicle's longitudinal axis and the distance to the device are determined in order to calculate and display the optimal position of the calibration device.
[0012] Disclosure of the invention
[0013] An inventive method for determining the position of a front camera of a motor vehicle in relation to at least one image acquisition device comprises optically capturing an area of the motor vehicle to be evaluated by at least one image acquisition device and calculating a 3D point cloud of the area to be evaluated. The following steps are performed:
[0014] -Identifying at least one first surface in at least one 2D image of the area to be evaluated (31) using an image-based segmentation algorithm,
[0015] -Identifying a second area within the first area using an image-based segmentation algorithm, wherein the second area represents an area of the front camera (18);
[0016] -Determination of the position of the front camera (18) in the 3D reference system of the at least one image recording device (6a, 6b) by projecting at least one reference point of the second surface from the 2D image into the 3D point cloud.
[0017] The use of an image-based segmentation algorithm increases the flexibility and robustness of the method. The segmentation algorithm can be trained on any geometric object and is therefore more flexible than methods based on predefined targets or edges. The method according to the invention is more robust to variations in lighting, textures, and minor damage or soiling on the vehicle. In particular, the method is less sensitive to such variations because it does not rely on the precise detection of R.417431.
[0018] - 3 - relies on specific features as in target-based methods, but on the segmentation of characteristic areas.
[0019] Unlike target-based methods, this eliminates the need to place and measure special reference objects. This saves time and money and simplifies the calibration process. The risk of errors due to incorrect target placement is also eliminated.
[0020] The entire process of object identification and location determination can be automated, making the calibration process more efficient and reproducible, and reducing human error.
[0021] By combining information from the 2D images with the data from the 3D point cloud, the position of a first and second surface in the reference system of the image acquisition device can be precisely determined, leading to more accurate calibration.
[0022] The image-based segmentation algorithm can recognize surfaces, especially a first and second surface, of different vehicle types in previously unknown images, making the method universally applicable.
[0023] Since no targets are required, calibration is possible in various environments without any special environmental requirements. Furthermore, the space requirement is minimal, allowing calibration even in small rooms or open areas.
[0024] The invention also includes a device for determining the position of a front camera of a motor vehicle in relation to at least one image acquisition device. The device comprises at least one image acquisition device for optically capturing an area of the motor vehicle to be evaluated, wherein the area to be evaluated comprises a windshield of the motor vehicle, and an evaluation unit configured to receive image data from the at least one image acquisition device and to perform the steps according to the invention.
[0025] The device according to the invention has the same advantages as the method according to the invention. R.417431
[0026] - 4 -
[0027] The dependent claims describe advantageous embodiments and further developments of the method and device according to the invention.
[0028] It is advantageous if the first surface is a trapezoidal surface arranged around the front camera, since the trapezoidal surface around the front camera usually stands out well from the surrounding area due to its bright coloring.
[0029] It is advantageous if the second surface is a circular surface representing the lens of the front camera, as this is easy to identify within an image-based segmentation algorithm.
[0030] Identifying an additional surface, representing the windshield, before identifying the first surface is advantageous because it leads to greater accuracy in the process. Only the first surfaces within the windshield are searched for, thus covering a smaller area than if the entire 2D image were considered.
[0031] Performing a plausibility check step, which rejects a result of identifying a first or second surface if a condition is not met, increases the accuracy of the procedure.
[0032] It is advantageous if the condition for the location of the second surface is that it is located in the upper half of the first surface, as this allows for the exclusion of incorrectly identified second surfaces, especially other circular surfaces, on and in the motor vehicle.
[0033] It is advantageous if the condition for the first surface is that it is located in the upper third of the further surface, especially the windshield, since the front camera is always located in an upper area of the windshield.
[0034] The use of a deep learning method for the image-based segmentation algorithm is advantageous, stemming from its ability to independently learn complex patterns and features in the image data. In contrast to R.417431
[0035] - 5 - In contrast to classical segmentation methods based on handcrafted rules, a deep learning model can automatically extract the relevant features for identifying geometric objects by training it with large datasets.
[0036] The advantage of using a "Segment Anything Model" (SAM) as an image-based segmentation algorithm lies in its remarkable ability to segment any objects in an image.
[0037] It is advantageous to determine the position of a calibration device in front of the vehicle based on the position of the front camera, because this allows for efficient and accurate calibration.
[0038] A further advantage arises when the size and / or orientation of an optical pattern / target on a calibration target is adjusted based on the position of the front camera, as the requirements for the alignment of the calibration target are lower, since deviations from the optimal alignment can be compensated for by the size and / or orientation of the optical pattern / target.
[0039] An embodiment of the invention is described below with reference to the accompanying drawings.
[0040] Brief description of the characters
[0041] Figure 1 shows a schematic top view of a measuring station with a motor vehicle and a calibration device;
[0042] Figure 2 shows a representation of a windshield with a front camera above the rearview mirror;
[0043] Figure 3 shows a schematic representation of an area to be evaluated and a windshield identified using an image-based segmentation algorithm;
[0044] Figure 4 shows a first surface, a second surface, and another surface identified using an image-based segmentation algorithm; and R.417431
[0045] - 6 -
[0046] Figure 5 schematically shows the determination of a surface representing the area of the front camera from a 3D point cloud and a segmented 2D image.
[0047] Character description
[0048] Figure 1 shows a schematic top view of a measuring station 2 with a motor vehicle 4 equipped with a driver assistance system 16. The driver assistance system 16 has at least one front camera 18.
[0049] The front camera 18 is installed in the motor vehicle 4 behind a windshield 41 and is positioned so that it looks forward through the windshield 41. The position of the front camera 18 behind the windshield 41 can vary depending on the vehicle type, but is usually located near a rearview mirror 15.
[0050] Figure 2 shows an exemplary embodiment of the mounting of the front camera 18 in a housing above a base 13 of the rearview mirror 15. The rearview mirror 15 is typically attached to the windshield 41.
[0051] Between the lens of the front camera 18 and the windshield 41, there is usually a cavity 11. The surface of this cavity 11 is light, in particular white, while the surrounding area 17 of the housing and / or the base 13 of the rearview mirror 15 is dark, in particular black. This contrast between the light cavity 11 and the darker surrounding area 17 results in a bright, high-contrast area around the lens of the front camera 18 appearing in the 2D image when viewing a 2D image of the windshield 17. This bright, high-contrast area around the lens of the front camera 18 stands out as a trapezoidal area 19 (thick border area of the cavity) against the darker surrounding area 17. This trapezoidal area 19 can be used for object detection by the front camera 18.
[0052] In alternative embodiments, the front camera 18 can also be integrated directly into the housing of the rearview mirror 15 or arranged in a separate housing next to the rearview mirror 15. R.417431
[0053] - 7 -
[0054] In a further embodiment, the method according to the invention can also detect a different geometry as the first surface around the lens of the front camera 18 instead of the trapezoidal surface 19.
[0055] A calibration device 20 is positioned in front of the motor vehicle 4.
[0056] The calibration device 20 shown in Figure 1 comprises a central calibration target 12. On the calibration target 12, for example, an optical pattern / target is formed, which is not visible in Figure 1 and which can be optically detected by the front camera 18 of the driver assistance system 16 in order to calibrate the front camera 18 and / or the driver assistance system 16.
[0057] The calibration target 12 can also be configured as an electronic display device, wherein the electronic display device is configured to electronically display an optical pattern / target. Depending on the orientation of the motor vehicle 4, in particular the position of the front camera 18, the optical pattern / target can be enlarged, reduced, shifted or distorted on the electronic display device in order to ensure optimal alignment of the optical pattern / target with respect to the front camera 18.
[0058] The device 7 according to the invention for determining the position of the front camera 18 of a motor vehicle 4 in relation to at least one image recording device 6a, 6b comprises at least one image recording device 6a, 6b which is designed to optically detect the motor vehicle 4 and to calculate a 3D point cloud of an area 31 to be evaluated.
[0059] The at least one image recording device 6a, 6b can, for example, comprise a stereo camera system 9 with a first image recording device 6a and a second image recording device 6b, which makes it possible to record three-dimensional images, in particular a 3D point cloud, of the motor vehicle 4.
[0060] The first image acquisition device 6a and the second image acquisition device 6b can each be attached to the right and left of the calibration target 12 and rigidly connected to it at a known distance. R.417431
[0061] - 8 -
[0062] In an alternative embodiment, the stereo camera system 9 can be positioned next to the calibration device 20, so that the stereo camera system 9 has both the calibration device 20 and the motor vehicle 4 in its field of view.
[0063] Other embodiments are also possible in which a single camera is moved and a 3D point cloud of the motor vehicle 4 is generated using the structure-from-motion method.
[0064] The device 7 according to the invention further comprises an evaluation device 8 which can perform the steps of the method according to the invention described below.
[0065] According to the method according to the invention, the at least one image recording device 6a, 6b is arranged in front of the motor vehicle 4 in such a way that an area 31 of the motor vehicle 4 to be evaluated can be optically detected and a 3D point cloud can be calculated.
[0066] In addition to the 3D point cloud, a 2D image of the area 31 to be evaluated is also recorded and evaluated with the evaluation device 8.
[0067] In the 2D image, at least one first surface, in particular the trapezoidal surface 19, is identified using an image-based segmentation algorithm. The image-based segmentation algorithm analyzes the 2D image and classifies each pixel, or, if a coarser resolution is used, a subset of the pixels, within the area to be evaluated 31. Each pixel is assigned either membership in the segmented first surface, in particular the trapezoidal surface, or in the background. This results in a binary segmentation mask, which defines a precise boundary between the first surface and the rest of the 2D image.
[0068] A deep learning method was used to train the image-based segmentation algorithm. The training dataset consists of 2D images of different vehicles 4 from various perspectives, under different lighting conditions, and with varying backgrounds. For each 2D image, a precise segmentation mask was created for at least one first surface, in particular the trapezoidal surface 19, which separates it from other parts of the vehicle 4 and the background. R.417431
[0069] - 9 -
[0070] A pre-trained deep learning algorithm was then fine-tuned using the specialized dataset of the first surface, for example, the trapezoidal surface 19. During this process, the model's weights were adjusted to maximize recognition accuracy. The inputs during training consisted of the images and their corresponding segmentation masks. These segmentation masks must be defined by the user.
[0071] According to one embodiment of the invention, the image-based segmentation algorithm is based on a SAM (Segment Anything Model) algorithm. The SAM (Segment Anything Model) algorithm is an image segmentation model developed by Meta AI Research that aims to precisely segment any object in a 2D image, even if that object was unknown to the model during training.
[0072] After identifying at least one first surface, a second surface within the first surface is identified using an image-based segmentation algorithm. The second surface represents a surface of the front camera 18, specifically the surface of the lens of the front camera 18. Since the lens of the front camera 18 is circular, the image-based segmentation algorithm must identify a circular surface 18' within the first surface.
[0073] The image-based segmentation algorithm was trained to identify the second surface within the first surface using a dataset of 2D images showing trapezoidal surfaces 19 from different perspectives, under varying lighting conditions, and with different backgrounds. For each 2D image, a precise segmentation mask of at least one second surface, in particular the circular surface 18', was created.
[0074] Figure 3 shows an embodiment of the method according to the invention. First, the windshield 41 is captured as the area 31 to be evaluated in a 2D image. After the 2D image of the area 31 to be evaluated has been transferred to the evaluation device 8, the trapezoidal area 19 (within the dashed box) is identified in the image section (B) by an image-based segmentation algorithm. The image-based R.417431
[0075] - 10 -
[0076] The segmentation algorithm was trained to calculate the trapezoidal area.
[0077] 19 can be identified in the 2D image of the windshield 41.
[0078] Image section (B) shows the result of segmentation using the Segment Anything algorithm. The trapezoidal area 19 is highlighted in white, while the rest of the 2D image is shown in black.
[0079] In a further step, as shown in image section (C), the second area, in particular the circular area 18', within the first area, in particular the trapezoidal area 19, is identified by an image-based segmentation algorithm.
[0080] The area 31 to be evaluated, in particular the windshield 41, was also captured by the at least one image recording device 6a, 6b and a 3D point cloud of the area 31 to be evaluated was calculated.
[0081] After the position of the second surface, which represents a surface of the front camera 18, has been identified in the 2D image, the position of the front camera 18 in the 3D reference system of the at least one image recording device 6a, 6b is determined by projecting at least one reference point 14 of the second surface from the 2D image into the 3D point cloud.
[0082] This is done by assigning individual pixels in the 2D image that belong to the second surface to the corresponding pixels in the 3D point cloud. In this way, at least one reference point 14 of the second surface identified in the 2D image can also be identified in the 3D point cloud.
[0083] The position of the at least one reference point 14 in the reference system of the at least one image recording device 6a, 6b can be used to calibrate the motor vehicle 4, since the position of the front camera 18 relative to the image recording device 6a, 6b can be determined using the position of the at least one reference point in the 3D reference system of the image recording device 6a, 6b.
[0084] Figure 4 shows a 3D point cloud of the windshield 41 in a 3D reconstruction image 53. In this 3D reconstruction image 53, each pixel is assigned a gray value that contains distance information. Furthermore, in the image section (B), the first surface within the area to be evaluated 31 R.417431 was identified.
[0085] - 11 - identified using the image-based segmentation algorithm. In image section (C), the second area, a circular area, within the first area was identified using the image-based segmentation algorithm.
[0086] The center of the circular area 18' can be chosen as reference point 14.
[0087] The position of this reference point 14 in the 3 D reference system of the image acquisition device 6a, 6b can be determined by projection or overlapping of the image information from the 3D reconstruction image 53 and the segmented 2D image.
[0088] The position of the front camera 18 in the 3D reference system of the at least one image recording device 6a, 6b can be determined from the position of the at least one reference point 14 in the 3D reference system of the image recording device 6a, 6b.
[0089] Figure 5 shows the procedure in a further embodiment of the method according to the invention. Here, before identifying the first surface, a further surface, which can represent the surface of the windshield 41, is identified in the 2D image of the area 31 to be evaluated using the image-based segmentation algorithm. This surface is shown in image section (A). This further surface then represents the area 31 to be evaluated for identifying the at least one first surface, which is shown in image section (B). This first surface then represents the area for identifying the second surface, which is shown in image section (C).
[0090] In another embodiment, a plausibility check procedure can be used which rejects a result of identifying a first or second surface if a condition is not met.
[0091] A first condition that leads to the rejection of the identification result can be the position of the second surface. For example, it can be checked whether the second surface is located in the upper half of the first surface. If the second surface is not located in the upper half of the first surface, the result is rejected. R.417431
[0092] - 12 -
[0093] A second condition that leads to the rejection of the identification result can be the position of the first area. For example, it can be checked whether the first area is located in the upper third of the subsequent area. If the first area is not located in the upper third of the subsequent area, the result is rejected.
[0094] The position of the front camera 18 can be used to calibrate the motor vehicle 4, since a position of a calibration device 20 in front of the motor vehicle 4 can be determined based on the position of the front camera 18.
[0095] In a further alternative process step, the size and / or orientation of the geometric object / target on the calibration plate 12, in particular the electronic display device, can also be adjusted based on the position of the front camera 18.
[0096] The evaluation device 8 can be configured to receive 2D images and / or 3D point clouds from the at least one image acquisition device 6a, 6b and to evaluate the 2D images and / or 3D point clouds provided by the at least one image acquisition device 6a, 6b according to the method according to the invention. The result can be displayed on a display device 10, which may, for example, include a screen and / or a printer.
[0097] The 2D images and / or 3D point clouds can be transmitted wirelessly, e.g. via a WLAN or BluetoothO data connection, or via a wired connection from the at least one image acquisition device 6a, 6b to the evaluation device 8.
[0098] At least one image acquisition device 6a, 6b, which is not attached to the calibration device 20, can be arranged on or at the measuring station 2 in such a way that the calibration device 20 is in the field of view of the at least one image acquisition device 6a, 6b.
Claims
R.417431 - 13 - Patent claims 1. Method for determining the position of a front camera (18) of a motor vehicle (4) in relation to at least one image recording device (6a, 6b), wherein an area (31) to be evaluated of the motor vehicle (4) is optically detected by the at least one image recording device (6a, 6b) and a 3D point cloud of the area (31) to be evaluated is calculated, wherein the area (31) to be evaluated comprises a windshield (41) of the motor vehicle (4), characterized in that the following steps are carried out: Identifying at least one first surface in at least one 2D image of the area to be evaluated (31) using an image-based segmentation algorithm, Identifying a second area within the first area using an image-based segmentation algorithm, wherein the second area represents an area of the front camera (18); Determination of the position of the front camera (18) in the 3D reference system of the at least one image recording device (6a, 6b) by projection of at least one reference point (14) of the second surface from the 2D image into the 3D point cloud.
2. Method according to claim 1, wherein the first surface is a trapezoidal surface (19) arranged around the front camera (18).
3. Method according to claim 1, wherein the second surface is a circular surface (18') which represents the lens of the front camera (18).
4. Method according to claim 1, wherein, prior to identifying the first surface, a further surface representing the surface of the windshield (41) is identified in a 2D image of the area (31) to be evaluated using an image-based segmentation algorithm, and this further surface represents the area (31) to be evaluated for identifying the at least one first surface. R.417431 - 14 - 5. The method of claim 1, wherein a plausibility check is performed which rejects a result of identifying a first or second surface if a condition is not met.
6. The method according to claim 1, wherein a condition for the position of the second surface is that it is located in the upper half of the first surface.
7. Method according to claim 1, wherein a condition for the first surface is that it is located in the upper third of the further surface, in particular the windshield (41).
8. The method of claim 1, wherein the segmentation algorithm is based on a deep learning method.
9. A method according to any of the preceding claims, wherein the segmentation algorithm is based on a Segment Anything Model.
10. Method according to one of the preceding claims, wherein a position of a calibration device (20) in front of the motor vehicle (4) is determined based on the position of the front camera (18).
11. Method according to one of the preceding claims, wherein a size and / or an orientation of an optical pattern / target on a calibration target 12 is adjusted based on the position of the front camera (18).
12. Device for determining the position of a front camera (18) of a motor vehicle (4) in relation to at least one image recording device (6a, 6b), comprising at least one image recording device (6a, 6b) for optically capturing an area of the motor vehicle (4) to be evaluated, wherein the area to be evaluated (31) comprises a windshield (41) of the motor vehicle (4), and an evaluation unit (8) which is configured to receive image data from the at least one image recording device (6a, 6b) and to perform the steps according to at least one of claims 1 to 11.