Method for calibrating a vehicle camera

Abstract

The invention relates to a method for calibrating a vehicle camera (2) which is an interior camera, wherein a QR code (7) is displayed on a vehicle display (3), said QR code containing information about a predefined position of the vehicle camera (2) in a vehicle coordinate system and about a number, size and position of illumination sources of the vehicle camera (2) in the vehicle coordinate system, the QR code (7) is read by means of a mobile device camera (5), the illumination source and the vehicle display (3) are simultaneously captured by means of the mobile device camera (5), the mobile device (4) locates a QR code (7) on the vehicle display (3), a position of the QR code (7) and a position of the illumination source in a coordinate system of the mobile device (4) are determined, a position of the vehicle camera (2) in the coordinate system of the mobile device (4) is determined by means of the position of the illumination source in the coordinate system of the mobile device (4), and the coordinate system of the mobile device (4) is transformed to the vehicle coordinate system by means of the position of the QR code (7) in the coordinate system of the mobile device (4) and a position of the vehicle camera (2) in the vehicle coordinate system is determined from the position of the vehicle camera (2) in the coordinate system of the mobile device (4).

Description

[0001] Mercedes Benz Group AG

[0002] Procedure for calibrating a vehicle camera

[0003] The invention relates to a method for calibrating a vehicle camera.

[0004] As described in WO 2020 / 052873 A1, a calibration system and a calibration procedure for a vehicle detection device are known from the prior art. The calibration system includes a camera that can be assigned to a fixed point relative to the vehicle. It also has an optically detectable, machine-readable information carrier configured to provide camera-related or calibration system-related information. The calibration system also includes a control unit that is coupled to the camera and configured to process the camera-related or calibration system-related information.

[0005] Furthermore, US patent 2024 / 0104879 A1 discloses systems and methods for calibrating interior cameras of a vehicle using optical markers that are positioned at predetermined locations within the cameras' field of view during calibration. These optical markers can be in the form of QR codes and carry information that is used during the calibration process.

[0006] The invention is based on the objective of providing a method for calibrating a vehicle camera that is improved compared to the prior art.

[0007] The problem is solved according to the invention by a method for calibrating a vehicle camera with the features of claim 1.

[0008] Advantageous embodiments of the invention are the subject of the dependent claims.

[0009] In a method according to the invention for calibrating a vehicle camera, wherein the vehicle camera is designed, configured, and arranged in the vehicle as an interior camera for at least partially capturing the passenger compartment of the vehicle and, in particular, for capturing at least one vehicle occupant, a QR code is displayed on a vehicle display. This QR code contains information about a predetermined 3D position (i.e., three-dimensional position) of the vehicle camera in a vehicle coordinate system, as well as the number, size, and 3D position of the vehicle camera's light sources in the vehicle coordinate system. It also contains, for example, information about the vehicle, such as the vehicle model and an equipment variant. The QR code is read by means of a mobile device camera.The mobile device is in particular a mobile phone, especially a smartphone, or for example a tablet computer.

[0010] Using the mobile device's camera, both the vehicle's camera (at least one or more light sources) and the vehicle's display (or displays) are simultaneously captured. The mobile device then uses its camera to locate at least one QR code (or multiple QR codes) displayed on the vehicle's display (or displays).

[0011] Using known camera information from the mobile device camera, including in particular a focal length and a physical size of its image sensor, as well as using a known size of the at least one displayed QR code, which is for example also a component of the first read QR code, and the known size of the at least one light source or the multiple light sources, a 3D position of the at least one or each QR code in a coordinate system of the mobile device is determined and a 3D position of the at least one light source in the coordinate system of the mobile device is determined.

[0012] Using the determined 3D position of the at least one light source in the coordinate system of the mobile device and a known 3D position of the at least one light source in a camera coordinate system of the vehicle camera, a 3D position of the vehicle camera in the coordinate system of the mobile device is determined. Using the determined 3D position of the at least one QR code on the at least one vehicle display in the coordinate system of the mobile device and the known 3D position of the at least one QR code in the vehicle coordinate system of the vehicle, the coordinate system of the mobile device is transformed to the vehicle coordinate system, and from the determined 3D position of the vehicle camera in the coordinate system of the mobile device, a 3D position of the vehicle camera in the vehicle coordinate system is determined.

[0013] The determined 3D position of the vehicle camera within the vehicle coordinate system is then advantageously compared with the predefined position of the vehicle camera within the vehicle coordinate system to identify any deviations between the actual 3D position of the vehicle camera determined in the described manner and the predefined 3D position. This deviation can then be taken into account, for example, when evaluating recordings from the vehicle camera or used for adjusting the vehicle camera. For instance, if the deviation is too large, a notification is issued to the vehicle user recommending that the vehicle camera be adjusted at a workshop.

[0014] The QR code or QR codes displayed on the vehicle display (or at least one) for position determination in the manner described above may correspond to the initial QR code containing information on the specified 3D position of the vehicle camera in the vehicle coordinate system and on the number, size and 3D position of the vehicle camera's light sources in the vehicle coordinate system, or may be designed differently.

[0015] The process is advantageously carried out using a program, in particular an app, on the mobile device. This may involve data transmission from the mobile device to the vehicle and / or the vehicle camera, and / or data transmission from the vehicle and / or the vehicle camera to the mobile device. The data transmission from the mobile device to the vehicle and / or the vehicle camera serves, for example, to activate the vehicle's display (or display) and the respective QR code, and to activate the vehicle camera or its light source(s).

[0016] In one embodiment of the method, at least one QR code or several QR codes are additionally displayed on a mobile device display of the mobile device.

[0017] QR codes are displayed, which are / are captured by the vehicle camera. Using known camera information from the vehicle camera, in particular the focal length and physical size of its image sensor, and a known size of the at least one or each QR code displayed on the mobile device display, a 3D position of the at least one or each QR code is determined in a camera coordinate system of the vehicle camera. A transformation is then performed between the camera coordinate system and the coordinate system of the mobile device using the known 3D position of the at least one or each QR code displayed on the mobile device display, thereby determining the 3D position of the vehicle camera in the coordinate system of the mobile device. This is thus a second method for determining the 3D position of the vehicle camera in the coordinate system of the mobile device.This improves the positioning of the vehicle camera in the coordinate system of the mobile device and thus also the subsequent positioning of the vehicle camera in the vehicle coordinate system.

[0018] In one embodiment, the mobile device is moved successively to several positions, in each of which the mobile device camera simultaneously captures both the at least one light source or the multiple light sources of the vehicle camera and the at least one or the respective vehicle display. In each position of the mobile device, the 3D position of the vehicle camera in the vehicle coordinate system is determined in the manner described above. This improves the position determination of the vehicle camera.

[0019] As mentioned above as an alternative, the mobile device camera is advantageously used to simultaneously capture both the one or more light sources of the vehicle camera and multiple vehicle displays, each showing at least one QR code or multiple QR codes. This allows, for example, the compensation of installation tolerances of the vehicle displays.

[0020] As already mentioned above as an alternative, it is advantageous to display several QR codes in different display positions on at least one or each vehicle display.

[0021] As mentioned above as a variant, the vehicle camera, for example, has several light sources.

[0022] This method is particularly advantageous because vehicle cameras are being used for an increasing number of functions in vehicles. These functions include both comfort features and important safety functions. To implement these functions precisely, it is essential that the vehicle cameras are installed very accurately in the vehicle, or that changes in the camera position are detected and the system adapts accordingly.

[0023] Since a change in camera position over a long period can never be completely ruled out, it is advisable to recalibrate the vehicle camera over time. This means that a mechanism is needed which...

[0024] It can determine the 3D position of the vehicle camera in relation to the passenger compartment in which the vehicle camera is installed.

[0025] The solution described here employs such a method, utilizing a mobile device equipped with a camera and, advantageously, a mobile device display. A smartphone, for example, is one such mobile device. Furthermore, the vehicle camera requires at least one active light source, and the vehicle must be equipped with at least one vehicle display.

[0026] A particular advantage of the solution described here is that, with the at least one light source or the multiple light sources of the vehicle camera and the at least one vehicle display or the multiple vehicle displays, each showing one or more QR codes, a sufficient number of suitable points in the passenger compartment are provided for calibration, which are simultaneously not visible or at least not disturbing to vehicle users during normal operation of the vehicle, and in particular do not negatively affect the design of the passenger compartment.

[0027] Another advantage is that the described solution offers the possibility for laypersons without special prior knowledge to easily and quickly perform calibration with a mobile device, such as a smartphone, since no special hardware is required for the described solution, but existing hardware, such as a smartphone or other mobile device, as well as the vehicle display(s) that are already present in the vehicle, can be used to carry out the procedure.

[0028] For example, all that is required is appropriate software, in particular an app, on the mobile device.

[0029] The described method thus enables simple calibration of the vehicle camera with high accuracy, since the advantageous variation of the mobile device's position and the QR codes on the vehicle displays, as described above, allows for the use of a large number of calibration points. Additionally, the search for suitable calibration points in the passenger compartment, which remain unchanged even over years, is eliminated. In principle, this method also allows vehicle users to recalibrate their vehicle camera themselves using their own smartphone or other mobile device, provided they are given access to appropriate software / app.

[0030] An advantageous embodiment of the method is briefly explained below:

[0031] The vehicle displays a QR code on its screen, which advantageously contains information about the vehicle and thus about the correct, i.e., predetermined, position of the vehicle camera. This advantageously includes, in particular, information about the predetermined 3D position of the vehicle camera within the vehicle's coordinate system and about the number, size, and 3D position of the vehicle camera's light sources within the vehicle's coordinate system. This QR code on the vehicle display is scanned, i.e., read, by a mobile device, specifically by its camera. The mobile device now has the necessary information about the vehicle and the vehicle camera, in particular the number and predetermined 3D position of the light sources within the vehicle's coordinate system.

[0032] The mobile device's camera, which is advantageously located on the same side as the mobile device's display, is then used and pointed at the vehicle camera and its light sources. An image processing algorithm then detects the position of the (advantageously two) light sources. The mobile device is then moved away from the vehicle camera until at least one vehicle display (or multiple displays) comes into the field of view of the mobile device's camera. However, the vehicle camera's active light sources must also remain within the mobile device's field of view. Thus, the mobile device's camera simultaneously captures both the vehicle camera's light source (or multiple light sources) and the vehicle display (or multiple displays).The respective vehicle display will show at least one QR code or several QR codes.

[0033] For the subsequent calibration step, it is advantageous to have known points within the vehicle. Due to the vehicle's design, some points are known, although not entirely precisely due to manufacturing tolerances. The 3D position of each QR code on at least one vehicle display is known within the vehicle's coordinate system. Additionally, the vehicle camera knows the 3D position of its active light source(s) within its own camera coordinate system because it was designed as a single component. Typically, the image sensor and light source are mounted on the same circuit board. Furthermore, at least one QR code is also displayed on the mobile device's screen, and the mobile device knows the 3D position of its displayed QR code within its own coordinate system.

[0034] For calibration, it is advantageous that both the vehicle and the mobile device display the QR codes on their screens. The mobile device locates the QR codes on the vehicle displays, or at least the one QR code on the vehicle display, while the vehicle camera locates the QR code on the mobile device's display. Since both cameras have information such as the focal length and the physical size of their image sensors, as well as the size of the respective QR code and the active light sources, depth localization is also possible, and the following points can be determined accordingly:

[0035] - 3D position of the at least one or each QR code on the at least one or each vehicle display in the coordinate system of the mobile device,

[0036] - 3D position of at least one QR code on the mobile device display within the camera coordinate system of the vehicle camera, and

[0037] - 3D positions of the active light sources of the vehicle camera in the coordinate system of the mobile device.

[0038] There is thus the coordinate system of the mobile device, the camera coordinate system of the vehicle camera, and the vehicle coordinate system of the vehicle. The goal of the calibration is to determine the 3D position of the vehicle camera within the vehicle's coordinate system in order to compare it with the predefined 3D position of the vehicle camera within the vehicle's coordinate system. The 3D position of the light source(s) has already been determined within the coordinate system of the mobile device. Together with the known coordinates of this light source(s) within the camera coordinate system of the vehicle camera, the 3D position of the vehicle camera within the coordinate system of the mobile device is determined.

[0039] To further transform from the mobile device's coordinate system to the vehicle's coordinate system, the mobile device must have recognized at least one QR code on at least one of the vehicle's displays, and the vehicle must also know its coordinates within its own coordinate system. This results in the transformation from the mobile device's coordinate system to the vehicle's coordinate system. Together with the initial transformation from the vehicle camera's coordinate system to the mobile device's coordinate system, this determines the 3D position of the vehicle camera within the vehicle's coordinate system. To further improve the accuracy of the transformations, the following additional steps are advantageously included:

[0040] - The vehicle camera also determines a transformation between the camera coordinate system of the vehicle camera and the coordinate system of the mobile device by locating the QR code on the mobile device display.

[0041] - The mobile device is moved to various positions, where its camera captures the vehicle displays and the vehicle camera with their light sources. This allows the necessary transformations to be calculated for many different positions of the mobile device.

[0042] - QR codes are used on several different vehicle displays to calculate installation tolerances of the individual vehicle displays.

[0043] - QR codes are displayed in different positions on each vehicle display.

[0044] This results in a large number of different points that can be used to determine the transformations. This allows the 3D position of the vehicle camera within the vehicle coordinate system to be determined very precisely.

[0045] Exemplary embodiments of the invention are explained in more detail below with reference to a drawing.

[0046] This shows:

[0047] Fig. 1 schematically shows a passenger compartment of a vehicle and a method for calibrating a vehicle camera in this passenger compartment of the vehicle.

[0048] Figure 1 shows an exemplary schematic representation of a vehicle 1, more precisely of an interior, in particular passenger compartment, of the vehicle 1.

[0049] The vehicle 1 has a vehicle camera 2, which is designed, configured, and arranged in the vehicle 1 as an interior camera for capturing at least sections of the passenger compartment. Furthermore, in the illustrated example, the vehicle 1 has several vehicle displays 3. To carry out a method for calibrating the vehicle camera 2, a mobile device 4, for example, a smartphone, is arranged in the passenger compartment. This device has a mobile device camera 5 and, in the exemplary embodiment of the method shown in Figure 1, also a mobile device display 6 on the side of the mobile device camera 5.

[0050] Vehicle 1 has a vehicle coordinate system with a vehicle coordinate origin FKU.

[0051] The vehicle camera 2 has a camera coordinate system with a camera coordinate origin KKU.

[0052] The mobile device 4 has a coordinate system with a coordinate origin GKU.

[0053] In the exemplary embodiment of the method shown in Figure 1, the vehicle 1 displays a QR code 7 on at least one of the vehicle displays 3. This QR code advantageously contains information about the vehicle 1 and thus about the correct, i.e., predetermined, position of the vehicle camera 2, and advantageously, in particular, information about the predetermined 3D position of the vehicle camera 2 in the vehicle coordinate system of the vehicle 1 and about the number, size, and 3D position of the light sources of the vehicle camera 2 in the vehicle coordinate system of the vehicle 1. The vehicle camera 2 must have at least one active light source for the method to function. Advantageously, it has several, for example, at least two, active light sources. The light sources, not shown here, are arranged on the vehicle camera 2.

[0054] Using the mobile device 4, in particular its mobile device camera 5, this QR code 7 on the vehicle display 3 is scanned, i.e., read. The mobile device 4 now knows the necessary information about the vehicle 1 and the vehicle camera 2, in particular the number and predefined 3D position of the light sources in the vehicle coordinate system.

[0055] The mobile device camera 5 of the mobile device 4 is now used and pointed at the vehicle camera 2 with its light sources. An image processing algorithm then detects the position of the (advantageously two) light sources. The mobile device 4 is then moved away from the vehicle camera 2 until the vehicle displays 3 come into the field of view of the mobile device camera 5 of the mobile device 4, whereby the active light sources of the vehicle camera 2 also remain in the field of view of the mobile device camera 5, as shown in Figure 1. The mobile device camera 5 thus simultaneously captures both the at least one light source or the multiple light sources of the vehicle camera 2 and the vehicle displays 3 of the vehicle 1. At least one QR code 7 or multiple QR codes 7 are displayed on the respective vehicle display 3.

[0056] For the subsequent calibration step, it is advantageous that there are known points in vehicle 1. Due to the design of vehicle 1, some points are known, although not entirely precisely due to assembly tolerances. The 3D position of the respective QR code 7 on the respective vehicle display 3 within the vehicle coordinate system is known. Additionally, the vehicle camera 2 knows the 3D position of its active light sources within its own camera coordinate system because it was designed as a single component. Typically, the image sensor and light source are arranged on the same circuit board. Advantageously, as shown in Figure 1, at least one QR code 7 is also displayed on the mobile device display 6 of the mobile device 4, and the mobile device 4 knows the 3D position of its displayed QR code 7 within its own coordinate system.

[0057] For calibration purposes, both vehicle 1 and mobile device 4 advantageously display the QR codes 7 on their displays 3 and 6. Mobile device 4 locates the QR codes 7 on the vehicle displays 3, while vehicle camera 2 locates the QR code 7 on the mobile device display 6 of mobile device 4.

[0058] Since both cameras know 2, 5 pieces of information, such as the focal length and the physical size of their image sensor, and also the size of the respective QR code 7 and the active light sources, localization for depth is also possible and accordingly the following points can now be determined:

[0059] - 3D position of the at least one or respective QR code 7 on the at least one or respective vehicle display 3 in the coordinate system of the mobile device 4,- 3D position of the at least one QR code 7 on the mobile device display 6 of the mobile device 4 in the camera coordinate system of the vehicle camera 2, and

[0060] - 3D positions of the active light sources of the vehicle camera 2 in the coordinate system of the mobile device 4.

[0061] The goal of the calibration is now to determine the 3D position of the vehicle camera 2 in the vehicle coordinate system of vehicle 1, in order to align it with the specified

[0062] To compare the 3D position of the vehicle camera 2 in the vehicle coordinate system of vehicle 1. The 3D position of the light sources has already been determined in the coordinate system of the mobile device 4. Together with the known coordinates of these light sources in the camera coordinate system of the vehicle camera 2, the 3D position of the vehicle camera 2 in the coordinate system of the mobile device 4 is determined.

[0063] To further transform from the coordinate system of mobile device 4 to the vehicle coordinate system of vehicle 1, mobile device 4 has recognized at least one QR code 7 on at least one of the vehicle displays 3 of vehicle 1, and vehicle 1 also knows its coordinates in its vehicle coordinate system. This results in the transformation from the coordinate system of mobile device 4 to the vehicle coordinate system of vehicle 1, and together with the first transformation from the camera coordinate system of vehicle camera 2 to the coordinate system of mobile device 4, the 3D position of vehicle camera 2 in the vehicle coordinate system of vehicle 1 is determined.

[0064] This specific 3D position of vehicle camera 2 in the vehicle coordinate system of vehicle 1 can now be compared with the specified position of vehicle camera 2 in the vehicle coordinate system of vehicle 1 in order to identify any deviations that may exist.

[0065] To further improve the accuracy of the transformations, the following additional points are advantageously provided:

[0066] - The vehicle camera 2 also determines a transformation between the camera coordinate system of the vehicle camera 2 and the coordinate system of the mobile device 4 by locating the QR code 7 on the mobile device display 6 of the mobile device 4.

[0067] - The mobile device 4 is moved to different positions, where its mobile device camera 5 captures the vehicle displays 3 and the vehicle camera 2 with their light sources. This allows the required transformations to be calculated for many different positions of the mobile device 4.

[0068] - QR codes 7 are used on several different vehicle displays 3 to calculate installation tolerances of the individual vehicle displays 3.

[0069] -On each vehicle display 3, QR codes 7 are displayed at different positions. This results in a large number of different points which can be used to determine the transformations. This allows the 3D position of the vehicle camera 2 in the vehicle coordinate system to be determined very precisely.

Claims

Mercedes Benz Group AG Patent claims 1. Method for calibrating a vehicle camera (2) of a vehicle (1), characterized in that the vehicle camera (2) is designed, configured and arranged in the vehicle (1) as an interior camera for at least partially capturing a passenger compartment of the vehicle (1), wherein a QR code (7) is displayed on a vehicle display (3) of the vehicle (1), which contains information on a predetermined 3D position of the vehicle camera (2) in a vehicle coordinate system of the vehicle (1) and on a number, size and 3D position of light sources of the vehicle camera (2) in the vehicle coordinate system of the vehicle (1), - the QR code (7) is read using a mobile device camera (5) of a mobile device (4), - using the mobile device camera (5) simultaneously captures both the at least one light source of the vehicle camera (2) and the vehicle display (3), - the mobile device (4) locates at least one QR code (7) displayed on the vehicle display (3) using its mobile device camera (5), - using known camera information from the mobile device camera (5), a known size of the at least one displayed QR code (7) and the known size of the at least one light source, a 3D position of the at least one QR code (7) is determined in a coordinate system of the mobile device (4) and a 3D position of the at least one light source is determined in the coordinate system of the mobile device (4), - a 3D position of the vehicle camera (2) in the coordinate system of the mobile device (4) is determined using the determined 3D position of the at least one light source in the coordinate system of the mobile device (4) and a known 3D position of the at least one light source in a camera coordinate system of the vehicle camera (2), and - the coordinate system of the mobile device (4) is transformed onto the vehicle coordinate system using the determined 3D position of the at least one QR code (7) in the coordinate system of the mobile device (4) and the known 3D position of the at least one QR code (7) in the vehicle coordinate system of the vehicle (1), and a 3D position of the vehicle camera (2) in the vehicle coordinate system is determined from the determined 3D position of the vehicle camera (2) in the coordinate system of the mobile device (4), and - the determined 3D position of the vehicle camera (2) in the vehicle coordinate system is compared with the specified position of the vehicle camera (2) in the vehicle coordinate system.

2. Method according to claim 1, characterized in that at least one QR code (7) is additionally displayed on a mobile device display (6) of the mobile device (4), which is detected by the vehicle camera (2), wherein a 3D position of the at least one QR code (7) in the camera coordinate system of the vehicle camera (2) is determined using known camera information of the vehicle camera (2) and a known size of the at least one QR code (7) displayed on the mobile device display (6), and a transformation between the camera coordinate system and the coordinate system of the mobile device (4) is carried out using the known 3D position of the at least one QR code (7) displayed on the mobile device display (6) in the coordinate system of the mobile device (4), and thereby the 3D position of the vehicle camera (2) in the coordinate system of the mobile device (4) is determined.

3. Method according to any one of the preceding claims, characterized in that the mobile device (4) is moved successively into several positions in which, at the same time, both the at least one light source of the vehicle camera (2) and the vehicle display (3) are captured by means of the mobile device camera (5), and in the respective position of the mobile device (4) the 3D position of the vehicle camera (2) in the vehicle coordinate system is determined.

4. Method according to any one of the preceding claims, characterized in that the mobile device camera (5) simultaneously captures both the at least one light source of the vehicle camera (2) and several vehicle displays (3), each of which displays at least one QR code (7) is displayed.

5. Method according to any one of the preceding claims, characterized in that several QR codes (7) are displayed in different display positions on the at least one or respective vehicle display (3).

6. Method according to any one of the preceding claims, characterized in that the vehicle camera (2) has multiple light sources.

Images