Method for storing validation data for validating driver assistance systems
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2026-01-22
- Publication Date
- 2026-07-30
Smart Images
Figure EP2026051540_30072026_PF_FP_ABST
Abstract
Description
[0001] R. 417296
[0002] - 1 -
[0003] title
[0004] Method for storing validation data for the validation of driver assistance systems
[0005] State of the art
[0006] The invention relates to the validation of driver assistance systems. Modern driver assistance systems process a multitude of sensor data and are capable of performing complex functions based on this data, which involve (sometimes significant) intervention in the interface between the driver and the vehicle or its functions. Such functions can include, for example, emergency braking assistance systems that intervene when an obstacle appears in traffic and the driver fails to react appropriately.
[0007] The evaluation of sensor data for the execution of functions in driver assistance systems is often performed by highly complex software systems. Before such driver assistance systems can be deployed in the field (in the regular operation of motor vehicles), extensive validations of the respective driver assistance system, and especially of the driver assistance system's software, are regularly required. This also generally applies when only (relatively minor) modifications are made to already established driver assistance systems.
[0008] According to commonly applied standards, thousands of hours of regular ferry operation of a vehicle with a specific driver assistance function are required to grant approval for the use of the respective driver assistance function in the field.
[0009] Due to the large number of driver assistance functions currently in use and their complexity, this is hardly feasible in practice. Therefore, a common approach today is to validate driver assistance functions under specific conditions in simulation environments and / or by feeding stored validation data into the control unit under test. In such simulation environments or control unit test fixtures, a variety of sensor data (especially labeled data) are used to test a driver assistance system and its functions, and to verify whether it functions correctly according to the sensor data. R. 417296
[0010] - 2 -
[0011] to react correctly to the represented situation. Such sensor data, which can be used in simulation environments to test driver assistance systems, is also regularly referred to as validation data. Validation data typically must include original sensor data as recorded in a vehicle during a test drive. The requirements for the originality of such validation data are usually significantly higher than those for training data, which is "only" used for training driver assistance functions. Training data is often generated entirely synthetically, for example, by adjusting data using filters to create additional training datasets. The higher requirements for validation data compared to training data apply because validation is what grants approval for driver assistance functions to be used on public roads.Validation based on validation data is therefore also carried out in many cases to validate training of driver assistance functions or control units for driver assistance functions based on training data.
[0012] Validating the multitude of different driver assistance systems used on the market requires entire data centers full of stored validation data.
[0013] In principle, it would be highly desirable to reduce the effort required to store and provide such validation data if compression methods could be used for this data. Possible methods are described, for example, in documents DE 102019214587 A1, US 11,356,579 B2, or EP 3185555 A1.
[0014] However, it is not yet possible to use such compressed data for validating the function of driver assistance systems. To meet common standards for validating driver assistance systems, the data used must typically correspond bit-for-bit to what is provided in the vehicle or by the sensors of the test vehicles. Therefore, validation data can currently only be lossily compressed if it is also processed with lossy compression in the vehicle during field testing. In the vehicle, such compression is associated with disadvantages such as additional latency for compression and decompression, the lack of use of CRC checksums to ensure end-to-end data integrity, and the costs associated with compression in series production. R. 417296
[0015] - 3 -
[0016] Starting from this premise, the object of the present invention is to alleviate or at least partially solve the problems described with reference to the prior art.
[0017] In particular, a solution should be provided that makes it possible to achieve higher levels of compression of validation data and thus significantly reduce the effort required to store and provide validation data.
[0018] Disclosure of the invention
[0019] This document describes a method for storing validation data in a data storage system. The validation data is intended for the validation of driver assistance systems. Furthermore, the data storage system is configured to provide the validation data for the validation of at least one driver assistance system, wherein the validation data includes at least some data recorded by sensors in vehicles during test drives.
[0020] The procedure consists of the following steps:
[0021] a) Receiving raw sensor data;
[0022] b) Performing a compression of the raw sensor data to generate compressed sensor data using a lossy compression method;
[0023] c) Generating synthesized sensor data based on the compressed sensor data generated in step b); and
[0024] d) Performing a comparison of the sensor raw data with the synthesized sensor data, checking whether a deviation between the synthesized sensor data and the sensor raw data received in step a) lies within a predetermined tolerance range, wherein the tolerance range is chosen such that deviations are detected which lie within a signal noise caused by characteristics of at least one sensor with which sensor raw data were recorded, wherein the tolerance range is further chosen such that the distance between synthesized sensor data and the recorded reality is not greater than the distance between the sensor raw data received in step a) and the same reality; and R. 417296
[0025] - 4 -
[0026] e) Storing the compressed sensor data in the data storage system if, in step d), it was determined that the deviation is within the specified tolerance range.
[0027] The advantage of the method according to the invention lies in the possibility of obtaining sensor data as part of validation data, and in particular compressed sensor data, with significantly less effort than previously required, while exhibiting the same deviation and thus quality as raw sensor data typically acquired directly from at least one sensor. In this context, the aforementioned comparison regarding the tolerance range contributes in particular to ensuring that the sensor data contained in the validation data is of such high quality that, after being stored in the data storage system, it is suitable for validating driver assistance systems. It also meets the necessary legal requirements.
[0028] It is particularly preferred if the raw sensor data were recorded during test drives using at least one image, ultrasound, lidar, and / or radar sensor, preferably at least one camera. The advantage of this embodiment is that the described method is applicable to various types of sensors and thus to different types of raw sensor data. Furthermore, the raw sensor data from different sensor types can be used to collect and provide various types of data for validation, which overall improves the quality of the method and thus the quality of the generated compressed validation data.
[0029] The reception of raw sensor data in step a) describes, in particular, the process by which a device set up to carry out the described procedure receives the validation data or the raw sensor data from the sensor(s) used. The device can be carried in a vehicle used to carry out the described procedure. Further steps of the described procedure—in particular steps b), c), and d)—can also be carried out in such a device. Optionally, the generation of the raw sensor data with the sensor(s) can also be part of the procedure, so that the reception of raw sensor data in step a) can also include the generation of raw sensor data with at least one sensor. This raw sensor data can then be processed directly using the described procedure.In some implementation variants, it is also possible that the sensor raw data is available before step a)R. 417296.
[0030] - 5 -
[0031] were temporarily stored and then "received" from a temporary storage area in step a).
[0032] According to another embodiment, the validation data includes additional data recorded in the vehicle during test drives, parallel to the raw sensor data. Such data could include, for example, interior data about the driver's and / or the vehicle's occupants' condition. For instance, the data could be image data from an interior camera capturing the driver's level of concentration. By incorporating this additional data, the quality of the compressed validation data can be further improved.
[0033] Furthermore, it can be advantageous if, according to one embodiment, the tolerance range used in step d) is selected such that deviations are detected which lie within the shot noise of sensor data contained in the validation data received in step a). The term "shot noise" within the meaning of the invention can be understood as a specific noise pattern that arises from the conversion of photons into electrons in a sensor and changes the pixel value according to a Poisson distribution, the magnitude of which depends on the exposure intensity.
[0034] This embodiment ensures that the compressed sensor data is of sufficient quality to allow for the (later) generation of synthesized sensor data.
[0035] It is further advantageous if the at least one sensor for determining the sensor raw data has a characteristic according to which the at least one sensor switches between different exposure times per pixel depending on the incidence of light, wherein the tolerance range used in step d) is determined taking this characteristic into account.
[0036] This embodiment is based on the idea that the aforementioned tolerance range takes into account pixels with varying brightness or darkness and can therefore be adapted more precisely to the respective raw sensor data.
[0037] Furthermore, it is preferred if the tolerance range used in step d) is adhered to for each pixel of sensor data in the validation data. R. 417296
[0038] - 6 -
[0039] It is ensured that the sensor data synthesized from the validation data stored in step e) are just as close to reality as the original raw sensor data received in step a). This criterion ensures that the sensor data derived from the validation data stored in step e) can be considered a direct representation or equivalent of the raw sensor data, thus allowing them to be used for validating at least one driver assistance system. This is possible at any arbitrarily short and different time points, making the data usable for verifying safety-critical and / or AI-based driver assistance systems, which are subject to higher safety levels and requirements than "pure" comfort driver assistance systems.
[0040] A further advantage is that, by verifying compliance with the tolerance range for each pixel, the method can also be used in systems that are susceptible to even the smallest changes in noise behavior. This enables the aforementioned applicability of the method to safety-critical and / or AI-based driver assistance systems, making the determined and stored validation data usable for a wider range of systems.
[0041] Furthermore, it is preferred that at least steps a) to d) be carried out in a vehicle during a test drive to record raw sensor data for the creation of validation data. In addition, not only test drives but also regular driving, for example by employees who have company cars, can be used for the aforementioned creation of validation data. This increases the amount of available validation data that can be advantageously used for the validation of the at least one driver assistance system.
[0042] To store the validation data long-term, step e) involves sending the compressed sensor data, as part of the validation procedure, from a vehicle to a stationary data storage system for use in a test drive. Alternatively, the data can also be sent via a wireless interface. Preferably, however, the compressed validation data is sent on hard drives, which are removed from the vehicle and then transferred to the stationary data storage system. This stationary data storage system could, for example, be a stationary hard drive farm where the data is stored. R. 417296
[0043] - 7 -
[0044] It is still preferred if the lossy compression procedure carried out in step b) is two-part and comprises the following two sub-steps:
[0045] b1) Performing a reduction of noise components in raw sensor data to generate cleaned raw sensor data; and
[0046] b2) Compressing the raw sensor data using a lossless reversible compression method.
[0047] This enables data-reduced compression, which requires less storage space. Thus, the overall amount of storage space required is reduced while maintaining the same data quality.
[0048] Furthermore, it is preferred that in step d) a comparison protocol is created which documents that the specified tolerance range is adhered to, and that the comparison protocol is stored in step b) together with the validation data. The comparison protocol advantageously serves to demonstrate that the compressed validation data is suitable and approved for validating the at least one driver assistance system.
[0049] Furthermore, a procedure for validating a driver assistance system is specified, comprising the following steps;
[0050] a. Receiving compressed validation data comprising compressed sensor data from a data storage system;
[0051] b. Creating synthesized sensor data from the compressed sensor data, c. Performing tests to validate the driver assistance system based on the validation data and the synthesized validation data,
[0052] the validation data received in step a) were stored in the data storage system according to the procedure already mentioned above.
[0053] This section also describes a system for validating at least one driver assistance system, which is set up to carry out the procedure described above.
[0054] The advantages and preferred configurations listed with regard to the procedures are to be applied analogously to the system and vice versa. R. 417296
[0055] - 8 -
[0056] The solution presented here and its technical context are explained in more detail below with reference to the figures. It should be noted that the invention is not intended to be limited by the exemplary embodiments shown. In particular, unless explicitly stated otherwise, it is also possible to extract partial aspects of the situations explained in the figures and combine them with other components and / or findings from other figures and / or the present description. The figures show schematic and exemplary representations of:
[0057] Fig. 1 shows a described method for storing validation data; and
[0058] Fig. 2: a method for validating a driver assistance system using such validation data.
[0059] In the figures, identical or equivalent components are always represented with the same reference symbols.
[0060] Fig. 1 shows a block diagram of a described procedure for storing validation data. The procedure steps a), b) (or b1) and b2)), c), d) and e) are shown schematically in Fig. 1.
[0061] In this method, the real environment 1 is detected by a sensor 2. The sensor 2 can be, for example, an image sensor, an ultrasonic sensor, a lidar sensor, and / or a radar sensor. Preferably, the sensor 2 is a camera that captures image data of the real environment 1.
[0062] The real environment 1 recorded by sensor 2 is transmitted as raw sensor data 3 to a driver assistance system 4, where it is used for a driver assistance function 5. The raw sensor data 3 is also compressed 6 to be stored as compressed validation data 11 in a data storage system 12 of a data center 15 via a transmission interface 13. The transmission interface 13 can be a wireless interface. Alternatively, storage can also take place on the data storage system 12 without transmission via a wireless interface. For example, a hard drive containing data can be removed from a test vehicle and physically transported to a data center for reading into the data storage system 12. R. 417296
[0063] - 9 -
[0064] Before storage, a noise reduction process 7 is applied to the raw sensor data 3. The cleaned sensor data 19, resulting from the noise reduction 7, are then subjected to reversible lossless compression 8. Simultaneously, the cleaned sensor data 19 are fed to a comparator 9, where noise is added 24. The cleaned sensor data 19 are then compared with the raw sensor data, and a comparison protocol 10 is generated. The comparison protocol 10 indicates how closely the cleaned sensor data 19, to which noise has been added, corresponds to the originally acquired raw sensor data 3. In other words, the comparison protocol 10 indicates whether a predefined tolerance range has been maintained. The comparison protocol 10 is then also stored in the data storage system 12 of the data center 15.
[0065] In parallel, additional data 22 are transferred to the transmission interface 13, which is then stored as validation data 23 on the data storage system 12 in the data center 15. Alternatively, the additional data 22 can also be transported to the data center 15 in the form of a hard drive, as described above.
[0066] This method of storing validation data offers a better solution compared to the state of the art, both in terms of storage space and effort.
[0067] Fig. 2 shows a block diagram of a method for validating a driver assistance system using validation data obtained according to the method explained with reference to Fig. 1. Steps b. and c. of this method are marked in Fig. 4. Step a. essentially corresponds to the method for storing validation data explained above with reference to Fig. 13. Step a. is accordingly shown in Fig. 1.
[0068] For this purpose, the compressed validation data 11 stored in the data storage system 12 are subjected to noise addition 24 using synthesized noise 21. By adding the synthesized noise 21, the validation data 23 essentially correspond to validation data obtained by known methods, which are also always noisy. R. 417296
[0069] - 10 -
[0070] The validation data 23, processed in this way, are then fed to a situation simulation or control unit test bench 17 in the form of synthetic sensor data. The situation simulation or control unit test bench 17 serves to simulate or validate a driver assistance system 4. For example, emergency braking scenarios can be simulated and the driver assistance system 4 validated accordingly. The validation data 23 preferably consists of data, such as image data, containing scenarios in which emergency braking by and is to be initiated by the driver assistance system 4.
[0071] This simulation or control unit test bench then leads to a validation result 18, which indicates whether the validation of the driver assistance system 4 was successful or not and provides values for assessing the performance of the system.
Claims
R. 417296 - 11 - Claims 1. Method for storing validation data (23) in a data storage system (12), the validation data (23) are intended for the validation of driver assistance systems (4); wherein the data storage system (12) is configured to provide the validation data (23) for the validation of at least one driver assistance system (4); and wherein the validation data (23) include at least some data (3) recorded by sensors (2) in vehicles during test drives; comprising the following steps: a) Receiving raw sensor data (3); b) Performing compression (6) of the sensor raw data (3) to produce compressed sensor data (11) using a lossy compression method; c) Generating synthesized sensor data (20) based on the compressed sensor data (11) generated in step b); and d) Performing a comparison of the sensor raw data (3) with the synthesized sensor data (20), checking whether a deviation between the synthesized sensor data (20) and the sensor raw data received in step a) lies within a predetermined tolerance range, wherein the tolerance range is chosen such that deviations are detected which lie within a signal noise caused by properties of at least one sensor (2) with which sensor raw data were recorded, wherein the tolerance range is further chosen such that the distance between synthesized sensor data (20) and the recorded reality is not greater than the distance between the sensor raw data (3) received in step a) and the same reality; and e) Storing the compressed sensor data (11) in the data storage system (12) if it was determined in step d) that the deviation is within the specified tolerance range.
2. Method according to one of the preceding claims, wherein the sensor raw data (3) are processed with at least one image, ultrasound, lidar and / or R. 417296 - 12 - Radar sensor, preferably with at least one camera, while test drives were recorded.
3. Method according to claim 2, wherein the (23) comprise further data (22) recorded in the vehicle during test drives in parallel with the sensor raw data (3).
4. Method according to one of the preceding claims, wherein the tolerance range used in step d) is selected such that deviations are detected which are within a shot noise of the sensor raw data (3) received in step a).
5. Method according to claim 4, wherein the at least one sensor (2) for determining the sensor raw data (3) has a characteristic according to which the at least one sensor (2) switches between different exposure times per pixel depending on the incidence of light, wherein the tolerance range used in step d) is determined taking into account this characteristic.
6. Method according to one of the preceding claims, wherein compliance with the tolerance range used in step d) for each pixel of sensor raw data (3) in the validation data ensures that the sensor data (20) synthesized from the validation data (23) stored in step e) are as close to reality as the original sensor raw data (3) received in step a).
7. Method according to one of the preceding claims, wherein by verifying compliance with the tolerance range for each pixel, the use is also possible in systems that are susceptible to minute changes in noise behavior.
8. Method according to one of the preceding claims, wherein at least steps a) to d) are carried out in a vehicle during a test drive to record raw sensor data (3) for the creation of validation data.
9. Method according to one of the preceding claims, wherein step e) involves sending the compressed sensor data (11) from a vehicle to R. 417296 - 13 - includes conducting a test drive to a stationary data storage system (12).
10. Method according to one of the preceding claims, wherein the lossy compression method (6) performed (b) is two-part and comprises the following two sub-steps) b1) Performing a reduction (7) of noise components in sensor raw data (3) to generate cleaned sensor data (19); and b2) Compressing (8) the cleaned sensor data (19) using a lossless reversible compression method.
11. Method according to one of the preceding claims, wherein in step d) a comparison protocol (10) is created which records that the specified tolerance range is adhered to, wherein the comparison protocol is stored in step b) together with the validation data.
12. A procedure for validating a driver assistance system comprising the following steps; a. Receiving validation data (23) comprising compressed sensor data (11) from a data storage system (12); b. Creating synthesized sensor data (20) from the compressed sensor data (11), c. Performing tests to validate the driver assistance system (4) based on the validation data (23) and the synthesized sensor data (20), wherein the validation data (23) received in step a) were stored in the data storage system (12) according to a method according to one of the preceding claims.
13. System for validating driver assistance systems (4) configured to perform a method according to claim 12.