Method for the optically based time synchronisation of data acquisition using a test vehicle and data acquisition using at least one drone accompanying the test vehicle

The optically based time synchronization method using a binary-coded timer with LEDs ensures accurate alignment and synchronization of drone and vehicle data, addressing the challenges of rotational displacement and time delays for neural network training in autonomous driving systems.

WO2026234296A1PCT designated stage Publication Date: 2026-11-12ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/063664
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-05-11
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Achieving precise spatial and temporal synchronization between a drone's and a test vehicle's coordinate systems for accurate 'ground truth' data acquisition is challenging, particularly due to rotational displacement and time delays, which affect the alignment and accuracy of neural network training for autonomous driving systems.

Method used

An optically based time synchronization method using a binary-coded timer with optical reference signals, such as LEDs, is employed to synchronize data between the drone and test vehicle, ensuring high precision and low cost with minimal additional equipment, by detecting binary values through multiple light sources and verifying them using threshold comparisons.

Benefits of technology

This method enables precise time synchronization of camera images, improving the accuracy of 'ground truth' data acquisition and enhancing the training of neural networks for sensor data evaluation in autonomous vehicles, despite adverse environmental conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026063664_12112026_PF_FP_ABST
    Figure EP2026063664_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for the optically based time synchronisation of data acquisition using a test vehicle (1) and data acquisition using at least one drone (2) accompanying the test vehicle (1), comprising the following steps: a) detecting signals and / or features of the operation of the test vehicle (1) in video data recorded by a camera (3) of the at least one accompanying drone (2); and b) carrying out the time synchronisation on the basis of a binary coded value of a time counter using the signals and / or features detected in step a).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] R. 416408

[0002] - 1 -

[0003] Description

[0004] title

[0005] Method for optically based time synchronization of data acquisition with a test vehicle and data acquisition with at least one drone accompanying the test vehicle

[0006] State of the art

[0007] A fundamental challenge in the development of systems for highly automated and autonomous driving of motor vehicles is the design of sensor systems.

[0008] Such systems typically include neural networks used to analyze and evaluate sensor data captured by the vehicle's environmental sensors.

[0009] The development of such systems therefore requires, in particular, a very large amount of training data with which neural networks can be trained. Such data is also referred to as "labeled data." This means that certain information is known about the data, which a neural network should recognize independently of comparable data. Such data is needed for training neural networks because it allows information to be fed into them. For training neural networks to evaluate sensor data from a vehicle, so-called "ground truth" data is regularly required. This data shows where certain objects located in the vehicle's environment, and which can be detected by the vehicle's sensors, are actually located in a coordinate system that is referenced to the ground to describe the vehicle's surroundings. Vehicle sensors can Ob-R. 416408

[0010] - 2 -

[0011] Objects are always only detected from the respective perspective of the sensor on the vehicle.

[0012] Using “ground truth” data regarding such objects, neural networks can be trained to evaluate sensor data from such sensors and reliably recognize the position of objects in the ground-based coordinate system to describe the vehicle environment.

[0013] Obtaining such "ground truth" data is complex. One known approach involves using a drone to collect this data while a vehicle undergoes test drives, in order to gather data for training neural networks. This drone preferably hovers above the test vehicle at all times during the test drive. Ideally, the drone is equipped with a camera that recognizes the test vehicle, its surroundings, and the objects within those surroundings. Due to the overhead perspective from which such a camera views the vehicle and the objects, "ground truth" data can be acquired much more easily and accurately using such a drone and its camera, with less complex evaluation systems than those required for evaluating data from the vehicle's sensors.Positioning a drone at an adjustable distance above the vehicle makes it very advantageous to obtain training data in this way - especially if the training data to be obtained contains the "ground truth" data described above.

[0014] When using drones to obtain training data, it is generally necessary to determine the relative position between the drone and the test vehicle very precisely. This is particularly important for correctly aligning the drone's camera image within the test vehicle's coordinate system. Typically, an (undetected) rotational displacement of the drone relative to the vehicle is especially problematic for using the drone's camera images for training data. For example, an angular error of just 0.1 degrees can lead to relative (incorrect) displacements of up to 10 cm for objects located 50 meters from the test vehicle. [The following appears to be unrelated and possibly a separate text fragment: The detection of positions of objects in the vicinity of a vehicle in R. 416408]

[0015] - 3 -

[0016] However, achieving centimeter-level accuracy with in-vehicle sensors is a goal that can be reached through appropriate training of neural networks to evaluate sensor data from such sensors. Training data must therefore exhibit comparable or even higher accuracy.

[0017] To generate highly accurate reference data using video footage captured by a drone flying over a test vehicle, precise spatial and temporal synchronization of the drone's and the vehicle's coordinate systems is required. Data processing in the test vehicle and data processing in the drone can introduce time delays or shifts, making it difficult to correlate data generated by cameras on the drone with data obtained from environmental sensors on the vehicle.

[0018] Approaches are known for the spatial synchronization of the coordinate systems of the test vehicle and the drone, in which markers are placed on the test vehicle (especially on its roof) that can be detected in camera images / camera data recorded by a camera on the drone. Such markers can be used to determine the spatial orientation and position of the test vehicle in the camera images / camera data.

[0019] Time synchronization means, in particular, that it is possible to temporally assign camera images recorded with the camera on the drone to data recorded with sensors on the test vehicle.

[0020] The time synchronization between the test vehicle and the drone should meet the following requirements:

[0021] • High precision: This means that the time assignment should be as accurate as possible;

[0022] • Low costs: This means, in particular, that the hardware required for time synchronization in the test vehicle and the drone should be as cost-effective as possible;

[0023] • as little or no additional measuring equipment as possible in the drone;

[0024] andR. 416408

[0025] - 4 -

[0026] • as little or no additional measuring equipment as possible in the test vehicle;

[0027] • if possible, no influence of environmental conditions and / or geographical conditions on the possibility and accuracy of time synchronization.

[0028] A basic option for temporal synchronization would be a lidar system. Objects visible to both the test vehicle's sensors and the video data recorded by the drone's camera could initially be used for spatial synchronization. Once spatial synchronization is complete, temporal synchronization can also be performed based on the object's position as seen by the drone and the test vehicle at a given time.

[0029] It was also suggested to perform time synchronization using GNSS time signals received by both the drone and the test vehicle.

[0030] Markers on test vehicles, which are detected in the video data generated by cameras on the drone, have been proposed for the spatial synchronization of the measurement systems on the test vehicle and on the drone.

[0031] These methods have in common that, in addition to the sensor data obtained in the vehicle and the camera data obtained with the drone's camera, further data must be recorded in order to enable a temporal assignment of certain sensor data to certain camera data.

[0032] It is also known that time synchronization between multiple cameras is currently achieved by timestamping the individual frames or by using camera slates. The optically based time synchronization method (OBTS) uses a shared subframe-rate binary counter to determine the required time offset between the cameras. The current value of this counter can be displayed via LEDs visible to all cameras. OBTS can also be performed with cameras with different:

[0033] • Timestamp type (or no timestamp at all); R. 416408

[0034] - 5 -

[0035] • Resolution,

[0036] • Frame rate.

[0037] Disclosure of the invention

[0038] Based on this, one object of the present invention is to at least partially solve the problems described in connection with the prior art and, in particular, to achieve an improvement in the optical time synchronization of camera images. This object is achieved by the invention according to the features of the independent claims. Further advantageous embodiments are specified in the dependent claims, as well as in the description and, in particular, also in the description of the drawings. It should be noted that a person skilled in the art can combine the individual features in a technologically meaningful way and thus arrive at further embodiments of the invention.

[0039] A method for optically based time synchronization of data acquisition with a test vehicle and data acquisition with at least one drone accompanying the test vehicle is described. The method comprises the following steps:

[0040] a) Detection of signals and / or characteristics of the operation of the test vehicle in video data recorded with a camera of at least one accompanying drone;

[0041] b) Performing time synchronization based on a binary-coded value of a time counter using the signals and / or features detected in step a).

[0042] The method is generally based on the functional aspect that, for example, two cameras optically read the same reference binary counter with an unknown time offset to synchronize the images. The binary counter reading can be viewed as a linear trend line. This is based on the general equation for linear trend lines yi = Mix + bi and y2 = M2X + b2R. 416408

[0043] - 6 -

[0044] The required time offset t can be calculated by t = b2 - bi, where Mi » M2 applies.

[0045] In particular, time synchronization based on the binary coded value according to step b) ensures a secure and improved determination of the time counter and therefore enables improved time synchronization of the data and especially the recorded images.

[0046] According to one embodiment, the binary-coded value is optically binary-coded. The optically binary-coded value of the timer is advantageous with regard to the use of cameras, since the timer value can thus be detected by the cameras.

[0047] In one embodiment, optical reference signals are recorded by the at least one drone and used to verify the binary encoding. Each optical reference signal is assigned to a digit of the binary-coded value of the timer.

[0048] This embodiment ensures valid and simple verification of the binary-coded value that can be detected by the at least one drone. Due to the fact that an optical reference signal is assigned to a digit of the binary-coded value of the timer, fail-safe verification of each digit of the binary-coded value is guaranteed.

[0049] It can be advantageous if the optical reference signals are provided by at least two light sources, in particular LEDs. Light sources, and especially LEDs, are simple solutions for implementing the optical reference signals and, moreover, ensure that the reference signals can be detected by the at least one drone. In other words, the binary-coded timer, and thus also the light source for the reference signals, is preferably located in the test vehicle. The at least one drone can then, while flying at an altitude of approximately 5 m to 10 m above the test vehicle, detect the reference signals provided by the light sources, even under adverse conditions, for example, due to rain and / or fog. R. 416408

[0050] - 7 -

[0051] According to a further embodiment, the binary encoding is verified depending on whether a value of the light source of the first row of light sources falls below or exceeds a threshold, wherein the threshold is determined by the recorded reference signals, wherein a state in which the respective light source of the first row of light sources is switched on defines a binary value above the threshold, and a state in which the respective light source of the first row of light sources is switched off defines a binary value below the threshold.

[0052] The advantage of this embodiment is that the verification of the binary encoding is very simple, yet reliable.

[0053] In detail and according to a further embodiment, each digit of the binary-coded value of the time counter is verified as a logical "1" if the binary value is above the threshold, and as a logical "0" if the binary value is below the threshold.

[0054] In other words, the reference signals verify the value of the binary-coded timer. This ensures that at least one drone, or, if more than one drone is used to capture image data, all drones, can detect the timer value to enable precise time synchronization of the images. Specifically, if only one of the light sources associated with a digit of the binary-coded value is switched on, the binary value of the reference signal will be below the threshold. This means that the respective binary value of the associated digit of the timer is a logical "0". This can be particularly helpful if the optically binary-coded timer digit is not visible, or only partially visible, to at least one drone.However, if the reference signal is visible to at least one drone, the value of that particular position in the timer can be reproduced and verified by knowing the value of the reference signal. In this case, if at least one drone detects that the value of the reference signal is below the threshold, the value of that position in the binary-coded timer can only be a logical "0" at a later time. R. 416408.

[0055] - 8 -

[0056] Furthermore, it can be advantageous if, according to another embodiment, the optical reference signals are provided by a plurality of light sources, and at least two light sources are assigned to each digit of the binary-coded time counter in order to verify the assigned digit of the binary-coded time counter. This means that each digit of the binary-coded time counter is assigned an optical reference signal, wherein the optical reference signal comprises at least two light sources.

[0057] Preferably, the binary-coded time counter is designed as a series of light sources, such as LEDs, wherein each light source represents a digit of the binary-coded time counter.

[0058] The light sources are preferably installed on a circuit board. Such a circuit board can, for example, be attached to the vehicle in such a way that it is visible in the video data recorded by the camera of the at least one drone. The light sources (in particular the circuit board with the light sources) can be mounted on the exterior structure of the test vehicle, for example on the roof, on a hood, and / or at other positions on the vehicle. In particular embodiments, the light sources (in particular the circuit board with the light sources) can also be installed inside the test vehicle in such a way that they are visible to the camera of a drone. In particular, the light sources / the circuit board can be arranged below a window of the vehicle, which is oriented so that a drone flying above the vehicle can see it.In particular, the light sources / circuit board can be positioned below a windshield on a shelf inside the test vehicle.

[0059] The optical reference signals, in particular the at least two light sources, are also arranged on the circuit board, specifically below each light source. This allows each digit of the binary-coded value of the timer to be verified by the at least two light sources located directly below the respective digit to be verified. R. 416408

[0060] - 9 -

[0061] To determine the orientation and / or angle of at least one drone relative to the test vehicle, visual markers are used. In particular, ArUco markers are used. ArUco markers are a type of fiduciary marker (visual markers for tracking or positioning) used in machine vision applications. These markers are designed to be easily detected and decoded by algorithms, making them suitable for various tasks such as augmented reality, robotics, and camera calibration.

[0062] Also described is a data processing device comprising a processor adapted and / or trained to perform the described procedure.

[0063] Furthermore, a computer program product is described which includes instructions that, when the program is executed by a computer, cause the computer to perform the described procedure.

[0064] Furthermore, a computer-readable storage medium is described which contains instructions which, when executed by a computer, cause the computer to perform the procedure described above.

[0065] The invention and its technical context are explained in more detail below with reference to the figures. The figures show preferred embodiments, to which the invention is not limited. It should be noted in particular that the figures, and especially the proportions shown in the figures, are merely schematic. The figures show:

[0066] Fig. 1 shows a schematic representation of a drone observing a test vehicle; and

[0067] Fig. 2 shows a circuit board for carrying out the method according to the invention, wherein the circuit board can be placed in the test vehicle from Fig. 1. R. 416408

[0068] - 10 -

[0069] Fig. 1 shows a schematic representation of a drone 2 accompanying a test vehicle 1 and observing it from an observation position 17 above the test vehicle 1. The drone 2 has a camera 3 with a downward-facing field of view 16, which can generate camera data or video data in which the test vehicle 1 and other objects 18 in the vicinity of the test vehicle 1 are recognizable / visible. The camera data or video data obtained with the camera 3 on the drone 2 are stored in a drone coordinate system 14, which is shown schematically here. The camera data or video data can be used to create a reference model with which sensors on the test vehicle 1 can be calibrated or algorithms for evaluating the data from such sensors can be improved.Such sensors can be, in particular, environmental sensors that can monitor the surroundings of the test vehicle 1 and detect and track objects 18 in the vicinity of the test vehicle 1. Data obtained from environmental sensors 1 on the test vehicle 1 is stored in a vehicle coordinate system 13. To use the camera data from camera 3, a conversion from the vehicle coordinate system 13 to the drone coordinate system 14 and / or vice versa is required. To enable this conversion, the orientation and position of the vehicle coordinate system 13 and the drone coordinate system 14 relative to each other are necessary.

[0070] Such calibration is possible using a marker 7 on the test vehicle 1, which is identifiable in the camera data or video data of camera 3. This marker 7 primarily enables spatial synchronization of the vehicle coordinate system 13 and the drone coordinate system 14. Furthermore, to compare the data from camera 3 with the data from sensors on the test vehicle 1, temporal synchronization of the camera data or video data from camera 3 of the drone 2 with the data from the sensors of the test vehicle 1 is required. A novel, particularly efficient approach for this temporal synchronization is proposed here.

[0071] Fig. 2 shows a circuit board 20 with which the method according to the invention can be carried out and which is explained in more detail below. R. 416408

[0072] - 11 -

[0073] The circuit board 20 can, for example, be arranged in the test vehicle 1 for optically based time synchronization of data acquisition with the test vehicle 1 and data acquisition with at least one drone 2 accompanying the test vehicle 1.

[0074] The circuit board 20 therefore comprises a first row of light sources 22, in particular LEDs. The first row of light sources 22 is configured to optically encode a timer value in binary. The timer value can be used for time synchronization of the data. In Fig. 2, hatching within the circle symbolizing a light source indicates that this light source is switched on. Accordingly, four of the light sources in the first row of light sources 22 are switched on. Switched-on light sources represent a logical "1", and switched-off light sources represent a logical "0" for optically encoding the timer value in binary.

[0075] During a flyover of the test vehicle 1, the at least one drone 2, and in particular several drones 2, can each capture the first row of light sources 22 and the displayed binary-coded value of the timer in order to synchronize the data. The circuit board 20 is preferably arranged behind the windshield in the test vehicle 1.

[0076] However, it is sometimes difficult for the at least one drone to detect and record the respective binary-coded value of the timer, particularly due to sunlight. To reliably verify whether a particular light source in the first row of light sources 22 is switched on or off, the circuit board includes a second row of light sources 24 and a third row of light sources 26. Specifically, the light sources in the second row of light sources 24 and the third row of light sources 26 are LEDs. The number of light sources in the second row of light sources 24 and the third row of light sources 26 preferably corresponds to the number of light sources in the first row of light sources 22. For example, in Fig. 2, the first row of light sources 22 comprises thirteen light sources. Similarly, the second row of light sources 24 and the third row of light sources 26 each comprise thirteen light sources.The light sources in the second row of R. 416408.

[0077] - 12 -

[0078] Light sources 24 and in the third row of light sources 26 are preferably of the same type as the light sources in the first row of light sources 22.

[0079] The second row of light sources 24 and the third row of light sources 26 are configured to provide reference signals for verifying the binary-coded value of the timer represented by the first row of light sources 22. For this purpose, one light source from the second row of light sources 24 and one from the third row of light sources 26 are each assigned to a light source from the first row of light sources 22. In other words, the first light source—viewed from the left side of the board—of the second row of light sources 24 and the first light source—viewed from the left side of the board—of the third row of light sources 26 are assigned as a reference signal pair to the first light source—viewed from the left side of the board—of the first row of light sources 22.Each light source in the reference signal pair emits a reference signal used to verify the respective switching state of the corresponding light source in the first row of light sources 22. Thus, for each light source in the first row of light sources 22, there is a pair of light sources with one light source deactivated in the second row of light sources 24 and one light source activated in the third row of light sources 26. This pair of light sources is located near the light source that encodes the binary value "0" or "1". The brightness of this light source in the video data acquired by the camera can be compared with the brightness of this light source pair in the video data. The pair of light sources from the second and third rows forms an individual reference value for the respective light source in the first row.

[0080] The binary encoding is verified depending on whether a value of the light source of the first row of light sources 22 falls below or exceeds a threshold, the threshold being determined by the recorded reference signals. A state in which the respective light source of the first row of light sources 22 is switched on defines a binary value above the threshold, and a state in which the respective light source of the first row of light sources 22 is switched off defines a binary value below the threshold. The entire second row of light sources 24 is switched off, and the entire third row of light sources 26 is switched on. R. 416408

[0081] - 13 -

[0082] Additionally, each digit of the binary-coded value of the timer is verified as a logical "1" if the binary value is above the threshold, and as a logical "0" if the binary value is below the threshold. In other words, and as can be seen from Fig. 2, the left light source—viewed from the left side of the board—of the first row of light sources 22 is off, which represents a logical "0" with respect to this digit in the binary-coded value of the timer. The first light source of the second row of light sources 24 and the first light source of the third row of light sources 26 form the associated reference signal pair of the first light source of the first row of light sources 22. The first light source of the second row of light sources 24 is off, while the first light source of the third row of light sources 26 is on.This means that the binary value of this reference signal pair is below the threshold. By additionally recording the switching state of the first light sources of the second row of light sources 24 and the third row of light sources 26, the value of the first light source of the first row of light sources 22 can be easily verified to avoid an error.

[0083] If, however, the third light source – viewed from the left side of the circuit board – of the first row of light sources 22 is considered, this light source is switched on. This switching state represents a logical "1" with respect to this position in the binary-coded value of the timer. As can also be seen in Fig. 2, the third light source of the second row of light sources 24 is switched off, and the third light source of the third row of light sources 26 is switched on. This means that the binary value based on this reference signal pair is above the threshold, indicating that the binary value of the respective position of the binary-coded timer must be a logical "1".By additionally recording the switching state of the third light sources of the second row of light sources 24 and the third row of light sources 26, the value of the third light source of the first row of light sources 22 can be easily verified to avoid an error.

[0084] This method allows each light source of the first row of light sources 22 to be verified, so that all digits of the binary-coded value of the timer can be easily determined. R. 416408

[0085] - 14 -

[0086] Additionally, the circuit board 20 also includes visual markers 28 for determining an orientation and / or an angle of the at least one drone 2 relative to the test vehicle 1. The visual markers 28 are preferably ArUco markers.

[0087] The features of the invention disclosed in the present description, the figures, and the claims can be essential for realizing the invention in its various embodiments, both individually and in any combination. The invention can be varied within the scope of the claims and taking into account the expert knowledge of a person skilled in the art.

Claims

R. 416408 - 15 - Claims 1. Method for optically based time synchronization of data acquisition with a test vehicle (1) and data acquisition using at least one drone (2) accompanying the test vehicle (1), comprising the following steps: a) Detection of signals and / or features of the operation of the test vehicle (1) in video data recorded by a camera (3) of the at least one accompanying drone (2); and b) Performing time synchronization based on a binary-coded value of a time counter using the signals and / or features detected in step a).

2. Method according to claim 1, wherein the binary-coded value is optically binary-coded.

3. Method according to claim 1 or 2, wherein optical reference signals are recorded by the at least one drone (2) which are used for verification of the binary encoding, wherein each optical reference signal is assigned to a digit of the binary encoded value of the time counter.

4. The method of claim 3, wherein the optical reference signals are provided by at least two light sources, in particular LEDs.

5. Method according to claim 4, wherein the binary encoding is verified depending on whether a respective light source of a first row of light sources (22) falls below or exceeds a threshold value, wherein a state in which the respective light source of the first row of light sources (22) is switched on defines a binary value above the threshold value, and a state in which the respective light source of the first row of light sources (22) is switched off defines a binary value above the threshold value. R. 416408 - 16 - When the light sources (22) are switched off, a binary value below the threshold is defined.

6. Method according to claim 5, wherein a respective digit of the binary coded value of the time counter is verified as a logical “1” if the binary value is above the threshold, and is verified as a logical “0” if the binary value is below the threshold.

7. Method according to any one of claims 4 to 6, wherein the optical reference signals are provided by a plurality of light sources (24, 26) and wherein at least two light sources are assigned to a digit of the binary coded time counter in order to verify the assigned digit of the binary coded time counter.

8. Method according to any of the preceding claims, wherein visual markers (28) are used to determine an orientation and / or angle of the at least one drone (2) relative to the test vehicle (1).

9. Device for processing data, comprising a processor adapted / configured to perform the method according to any one of claims 1 to 8.

10. Computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.

11. Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 8.