Method for processing raw sensor data in order to generate validation data

Abstract

The invention relates to a method for processing raw sensor data (3) which are recorded by means of at least one sensor (2) in a vehicle on a test journey in order to obtain validation data for validating driver assistance systems (4), comprising at least the following steps: A) receiving the raw sensor data (3) from at least one sensor (2); B) reducing a noise component from the raw sensor data (3) in order to generate adjusted sensor data (19); and C) providing the adjusted sensor data (19) for storage.

Description

[0001] R. 417530

[0002] - 1 -

[0003] title

[0004] Methods for processing raw sensor data to generate validation data

[0005] State of the art

[0006] The invention relates to the validation of driver assistance systems. Modern driver assistance systems process a multitude of sensor data and are capable of performing complex functions based on this data, which involve (sometimes significant) intervention in the interface between the driver and the vehicle or its functions. Such functions can include, for example, emergency braking assistance systems that intervene when an obstacle appears in traffic and the driver fails to react appropriately.

[0007] The evaluation of sensor data for the execution of functions in driver assistance systems is often performed by highly complex software systems. Before such driver assistance systems can be deployed in the field (in the regular operation of motor vehicles), extensive validations of the respective driver assistance system, and especially of the driver assistance system's software, are regularly required. This also generally applies when only (relatively minor) modifications are made to already established driver assistance systems.

[0008] According to commonly applied standards, thousands of hours of regular ferry operation of a vehicle with a specific driver assistance function are required to grant approval for the use of the respective driver assistance function in the field.

[0009] Due to the large number of driver assistance functions in use today and the complexity of these functions, this is hardly feasible in practice. Therefore, a common approach today is to validate driver assistance functions under specific conditions in simulation environments and / or by feeding stored validation data into the control unit under test. In such simulation environments or control unit test fixtures, a variety of (especially labeled) sensor data are used to test a driver assistance system and to verify whether it reacts correctly in the situation represented by the sensor data. Such sensor data, which can be used in simulation environments, are described in R. 417530.

[0010] - 2 -

[0011] The data used to test driver assistance systems is also regularly referred to as validation data. Such validation data typically needs to include original sensor data as recorded in a vehicle during a test drive.

[0012] Typically, the requirements for the originality of validation data are significantly higher than those for training data used "only" for training driver assistance functions. Training data is often generated entirely synthetically, for example, by filtering data to create additional training datasets. The higher requirements for validation data compared to training data stem from the fact that validation grants approval for driver assistance functions to be used on public roads. Therefore, validation based on validation data is often performed to validate training of driver assistance functions or control units for driver assistance functions, which is itself based on training data.

[0013] Validating the multitude of different driver assistance systems used on the market requires entire data centers full of stored validation data.

[0014] In principle, it would be highly desirable to reduce the effort required for storing and providing such validation data if compression methods could be used for this data. Possible compression methods and approaches are described, for example, in documents DE 102019214587 A1, US 11,356,579 B2, and EP 3185555 A1.

[0015] However, it is not currently possible to use such compressed data for validating the functionality of driver assistance systems. To meet common standards for validating driver assistance systems, the data used must typically be bit-identical to what is provided in the vehicle or by the sensors of the test vehicles. Therefore, validation data can currently only be lossily compressed if it is also processed with lossy compression in the vehicle during field testing. In the vehicle, such compression has disadvantages such as additional latency for compression and decompression, the lack of use of CRC checksums to ensure end-to-end data integrity, and the associated costs for compression in series production. R. 417530

[0016] - 3 -

[0017] Starting from this premise, the object of the present invention is to alleviate or at least partially solve the problems described with reference to the prior art.

[0018] In particular, a method should be specified that makes it possible to achieve higher levels of compression of validation data and thus significantly reduce the effort required to store and provide validation data.

[0019] Disclosure of the invention

[0020] This document describes a method for processing raw sensor data recorded in a vehicle during a test drive to obtain validation data for validating driver assistance systems with at least one sensor. The method comprises at least the following steps:

[0021] A) Receiving raw sensor data from at least one sensor;

[0022] B) Reducing the noise component from the raw sensor data to generate cleaned sensor data; and

[0023] C) Providing the cleaned sensor data for storage.

[0024] Preferably, the following step can be performed after step A):

[0025] a1) Processing of the raw sensor data in a driver assistance system for a driver assistance function.

[0026] The reception of raw sensor data in step A) describes, in particular, the process by which a device set up to carry out the described method receives the raw sensor data from the sensor(s) used. The device can be carried in a vehicle used to carry out the described method. Further steps of the described method can also be carried out in such a device – in particular steps B) and C). Optionally, the generation of the raw sensor data with the sensor(s) can also be part of the method, so that the reception of raw sensor data in step A) can also include the generation of raw sensor data with at least one sensor. This raw sensor data can then be processed directly using the described method. In embodiments, it is also R. 417530

[0027] - 4 -

[0028] It is possible that the raw sensor data was temporarily stored before step A) and then "received" from a buffer in step B).

[0029] In parallel with processing the raw sensor data to obtain validation data, the raw sensor data can also be used directly in the test vehicle itself to execute driver assistance functions. This corresponds to step a1) described above.

[0030] The term "test drive" can refer to two things: firstly, a test drive deliberately defined with a specific goal and / or parameters; and secondly, it can also encompass normal driving, for example, by employees using company cars. This increases the amount of available validation data that can be used to advantage in validating at least one driver assistance system.

[0031] The invention is based on the idea that the data used to date for validating the function of driver assistance systems must correspond bit-for-bit to what occurs in the vehicle. Therefore, the data can only be lossily compressed if it is also processed in the vehicle using lossy compression. In the vehicle, such compression is associated with disadvantages such as additional latency for compression and decompression, the lack of use of CRC checksums to ensure end-to-end data integrity, and the costs associated with mass production for compression.

[0032] By dividing the lossy compression into a part that includes the permanent changes in the image (namely the reduction of the noise component according to step B)) and a lossless part (a subsequent lossless compression), the later validation can be carried out with lossily compressed images, thus achieving significant cost savings without the disadvantages of data compression in vehicles in the field.

[0033] The core idea of ​​the invention is therefore not to initially perform complete lossy compression and decompression in the vehicle, but only to carry out the modifications to the raw sensor data as they result from the combination of R. 417530.

[0034] - 5 -

[0035] Both steps are performed. This is significantly less complex and results in less latency.

[0036] Furthermore, it is preferred if the following process steps are carried out after step B) and before step C):

[0037] c1) Generating synthetic sensor data from the cleaned sensor data by adding a synthetic noise component,

[0038] c2) Performing a comparison of synthetic sensor data with the raw sensor data received in step a), checking whether any deviation between the synthetic sensor data and the raw sensor data received in step a) is within a specified tolerance range,

[0039] where the provision of the corrected sensor data in step c) only takes place if it has been determined in step c2) that the deviation is within the specified tolerance range.

[0040] These two additional process steps ensure that the generated synthetic sensor data essentially corresponds to, and is therefore only as lossy as, the received raw sensor data. This ensures that the generated synthetic sensor data does not introduce any additional losses that would be detrimental to the validation of the driver assistance system. In other words, according to the embodiment described above, it is checked whether the generated synthetic sensor data is similarly lossy to the received raw sensor data in order to represent the generated synthetic sensor data as realistically as possible.

[0041] It is still preferred that in step c2) synthetic sensor data are generated only from a subset of the cleaned sensor data and that the comparison in step c2) is only made between the synthetic sensor data and a corresponding part of the raw sensor data.

[0042] The portion of the cleaned sensor data generated from the synthetic sensor data in step c2) can, for example, be a section of an image contained in the raw sensor data. This portion can also be a statistical representation of the raw sensor data, and / or at least one statistical value derived from / based on the raw sensor data. R. 417530

[0043] - 6 -

[0044] By using only a subset of the data to generate synthesized sensor data, the computational effort required to generate synthesized sensor data in step c1) can be reduced.

[0045] According to one embodiment, the at least one sensor from which raw sensor data is received in step a) is at least one image, radar, lidar, or ultrasonic sensor, in particular a camera. The raw sensor data is accordingly preferably image data, in particular camera data.

[0046] The method described here can be used, in particular, for processing raw sensor data that exhibits shot noise or shot noise artifacts. In the context of the invention, "shot noise" refers to a specific noise pattern that arises from the conversion of photons into electrons and alters the pixel value according to a Poisson distribution, with the magnitude of the change depending on the exposure intensity. The data can therefore also include data acquired from environmental and / or interior sensors of the vehicle. For example, the data could be image data from an interior camera that captures the driver's level of concentration. By incorporating this additional data, the quality of the compressed validation data can be further improved.

[0047] Furthermore, it is advantageous if, in step C), lossless reversible compression of the cleaned sensor data is carried out to prepare for storage of the cleaned sensor data.

[0048] This compression process preferably results in compressed sensor data from the cleaned sensor data. Preferably, the cleaned sensor data is provided for storage in step C) in the form of the compressed (cleaned) sensor data.

[0049] By compressing the cleaned sensor data, a significant reduction in storage space can be achieved compared to storing uncompressed data.

[0050] Furthermore, it is preferred if the noise reduction is performed individually for pixels or pixel groups. This embodiment of noise reduction allows for more accurate noise reduction per pixel or pixel group. R. 417530

[0051] - 7 -

[0052] This means that for individual pixel groups in the raw sensor data, which have different parameters (e.g., brightness) per group, a different level of noise reduction can or must be applied. This improves the overall quality of the cleaned sensor data.

[0053] In a preferred embodiment, in step B) noise components are reduced from the raw sensor data caused by characteristics of the at least one sensor from which the raw sensor data is received in step A). ​​This allows for a simple reduction of noise components and also provides a good basis for adding the aforementioned synthetic noise component. Since the noise component of the at least one sensor is usually very well determined or even known, such a noise component can be readily reproduced and thus added to the synthetic sensor data as synthetic noise.

[0054] Furthermore, it can be advantageous if reducing the noise component in step B) includes non-equidistant quantization of the sensor raw data.

[0055] With such non-equidistant quantization, it is particularly possible to take into account an unequal distribution of noise at the bit level in the raw sensor data.

[0056] This embodiment takes particular account of the fact that the noise in the sensor raw data is contained, especially, in the brightness values ​​of individual pixels. The sensor raw data contains a brightness value for each pixel, which is encoded, for example, with 16 bits and can therefore contain approximately 65,536 different brightness values. A light-dark boundary typically exists. Brightness values ​​below this light-dark boundary no longer carry any informational value. If the cutoff line is, for example, 5,000, then all brightness values ​​between 0 and 5,000 are equally considered "dark" or "black." Typically, the brightness values ​​immediately above the cutoff line have a high information content. Here, for example, 10 brightness values ​​(e.g., 5,001 to 5,010) can be represented by one bit. Much higher brightness values ​​usually have a much lower information value. For example...The brightness values ​​60,000 to 65,536 can also be mapped to one bit. In this case, 5,536 brightness values ​​would be mapped to one bit. This deviation in the mapping function is referred to as non-R. 417530.

[0057] - 8 -

[0058] equidistant quantization is described, whereby the numerical values ​​used here are only examples.

[0059] It is further preferred to use at least one lookup table to reduce the noise component in step B). This is preferably done by inputting signal segments into the lookup table and outputting simplified signal segments with less data.

[0060] The advantage of this embodiment is that lookup tables preferably represent a non-equidistant quantization of the raw sensor data. If, as already described, a noise component is added to the cleaned sensor data (see step c1), a reverse transformation is preferably performed using an inverse lookup table.

[0061] Furthermore, it is preferred if, before reducing the noise component in step B), information from several images in the sensor raw data is combined to represent a non-linear behavior of a sensor.

[0062] This embodiment advantageously addresses the fact that normal sensors are usually inherently somewhat lossy due to their design, because they internally create several images, e.g. four images, in order to then select the best pixel from these images and pass it on.

[0063] According to another preferred embodiment, the following step is carried out in addition to step c):

[0064] D) Storing cleaned sensor data in a data storage system for later provision as validation data for the validation of driver assistance systems.

[0065] This allows the data to be used later or multiple times.

[0066] The cleaned sensor data is stored, in particular, in the form of compressed sensor data.

[0067] It is further preferred if, in parallel to step a), additional data recorded in the vehicle is received and stored in the data storage system together with the cleaned sensor data in step d), in order to R. 417530

[0068] - 9 -

[0069] together with the cleaned sensor data, they can be provided as validation data for the validation of driver assistance systems.

[0070] Such data can include, in particular, additional data that is recorded in parallel with the raw sensor data and that can be used to label the raw sensor data.

[0071] It is also advantageous if meta-information contained in the raw sensor data is not affected by the removal of the noise component in step b) and is available for further processing of the sensor data.

[0072] For the purposes of this invention, the term "meta-information" can be understood to mean, in particular, an image number that represents an image contained in the sensor raw data. Such an image number can be encoded in the sensor raw data, for example, by using the last bits for storing the grayscale values ​​of specific pixels to encode the image number. In alternative embodiments, such meta-information can also be included in the sensor raw data as additional header or footer information.

[0073] The image number of the images can preferably be used in the generation of synthesized sensor data, in particular as a starting value C.Seed“ for a random noise generator, with which an artificial noise signal can be generated that can be added to the cleaned sensor data.

[0074] Using the image number, which is already present in the raw sensor data, as the "seed" value for random number generation offers significant advantages. This image number does not need to be generated separately, nor is an additional data source, such as a clock, required to obtain it. However, this image number is random relative to the raw sensor data (especially the image data) itself. At the same time, reproducibility is ensured because this image number is fixed for every image present in the raw sensor data. This can be helpful for the traceability of the analysis of the synthesized sensor data. R. 417530

[0075] - 10 -

[0076] Other data, particularly data relating to driving situations that may be used later, can include, but are not limited to:

[0077] Consecutive image numbering; and / or

[0078] Temperature measured by a temperature sensor; and / or exposure data; and / or

[0079] Bus data; and / or

[0080] The passenger sets labels; and / or

[0081] Images are manually evaluated by humans; and / or subsequent machine labeling; and / or

[0082] Use of very high-quality sensors that evaluate the "inferior standard sensors".

[0083] Furthermore, it is preferred if the following step is carried out before step D):

[0084] d1) Transmission of the data via a radio interface from the vehicle with at least one sensor from which the raw sensor data were determined in step a) to a stationary data storage system.

[0085] The invention further relates to a data processing system in a vehicle, which is equipped for carrying out the method described above.

[0086] Furthermore, the invention relates to a vehicle equipped for conducting test drives and collecting data for the creation of validation data, wherein the vehicle has the aforementioned

[0087] Data processing system included.

[0088] The advantages and preferred configurations listed with regard to the process are to be transferred analogously to the data processing system and to the vehicle, and vice versa.

[0089] The solution presented here and its technical context are explained in more detail below with reference to the figures. It should be noted that the invention is not intended to be limited by the exemplary embodiments shown. In particular, unless explicitly stated otherwise, it is also possible to extract partial aspects of the situations explained in the figures and combine them with other components. R. 417530

[0090] - 11 -

[0091] and / or insights from other figures and / or the present description are combined. They show schematically and by way of example:

[0092] Fig. 1 shows a described method for providing validation data;

[0093] Fig. 2: a method for validating a driver assistance system using such validation data;

[0094] Fig. 3: a transfer function for noise reduction in sensor data; and

[0095] Fig. 4. Comparison of raw sensor data and cleaned sensor data.

[0096] In the figures, identical or equivalent components are always marked with the same reference symbols.

[0097] Fig. 1 shows a block diagram of a described procedure for providing validation data. The procedure steps A), B), C), D) as well as a1), c1), c2) and d1) are each schematically assigned here.

[0098] In this method, the real environment 1 is detected by a sensor 2. The sensor 2 can be, for example, an image sensor, an ultrasonic sensor, a lidar sensor, and / or a radar sensor. Preferably, the sensor 2 is a camera that captures image data of the real environment 1.

[0099] The real environment 1 recorded by sensor 2 is transmitted as raw sensor data 3 to a driver assistance system 4, where it is used for a driver assistance function 5. The raw sensor data 3 is also compressed 6 to be stored as compressed validation data 11 via a radio interface 13 in a data storage system 12 of a data center 15. Alternatively, storage can also occur without transmission via the radio interface 13 on the data storage system 12, e.g., a hard drive. The data storage system 12 is then either brought to the data center 15 or physically shipped there.

[0100] Before storage, however, a noise reduction process 7 is applied to the raw sensor data 3. The data available after noise reduction 7 is R. 417530.

[0101] - 12 -

[0102] The cleaned sensor data 19 are then subjected to reversible lossless compression 8. The cleaned sensor data 19 are then fed to a comparator 9, where noise is added using a synthesis unit 24. Subsequently, the cleaned sensor data 19 are compared with the raw sensor data, and a comparison protocol 10 is generated. The comparison protocol 10 indicates how closely the cleaned sensor data 19, to which noise has been added, corresponds to the originally acquired raw sensor data 3. In other words, the comparison protocol 10 indicates whether a predefined tolerance range has been maintained. The comparison protocol 10 is then also stored in the data storage system 12 of the data center 15.

[0103] In parallel, additional data 22 are transmitted to the radio interface 13, which is then stored as validation data 23 on the data storage system 12 in the data center 15. Alternatively, the additional data 22 can also be transported to the data center 15 in the form of a hard drive, as described above.

[0104] This method of providing validation data enables a better solution compared to the state of the art, both in terms of storage space and effort.

[0105] Fig. 2 shows a block diagram of a method for validating a driver assistance system with such validation data obtained according to the method explained with reference to Fig. 1.

[0106] For this purpose, the compressed validation data 11 stored in the data storage system 12 are subjected to noise augmentation using synthetic noise via a synthesis unit 24. By adding the synthesized noise 21, the validation data 23 essentially correspond to validation data obtained by known methods, which are also always noisy.

[0107] The validation data 23, thus "processed," are then fed to a situation simulation or control unit test bench 17 in the form of synthetic sensor data. The situation simulation or control unit test bench 17 serves to simulate or validate a driver assistance system 4. Here, R. 417530

[0108] - 13 -

[0109] For example, emergency braking scenarios are simulated and the driver assistance system 4 is validated accordingly. The validation data 23 preferably consists of data, such as image data, containing scenarios in which emergency braking is to be initiated by and by the driver assistance system 4.

[0110] This simulation then leads to a validation result 18, which indicates whether the validation of the driver assistance system 4 was successful or not and provides metrics for assessing the performance of the system.

[0111] Fig. 3 shows a transfer function for noise reduction in sensor data.

[0112] The graph shown in Fig. 3 illustrates the transfer function 27 and its intervals for various input data 25 and output data 26. The transfer function 27 exhibits larger intervals when the noise is higher. This can be seen in the upper right section of Fig. 3.

[0113] In Fig. 4, raw sensor data 3, 28 - shown here on the left - and cleaned sensor data 19, 29 - shown here on the right - are shown in comparison to each other.

[0114] As can be seen in Fig. 4, a light-dark boundary 30 exists below which the brightness values ​​there no longer carry any informational value. The noise of the sensor raw data 3, 28 is contained in particular in the brightness values ​​of individual pixels of the sensor raw data 3, 28. The sensor raw data contains a brightness value for each pixel, which is encoded, for example, with 16 bits and can therefore contain approximately 65,536 different brightness values. If the cutoff value (30) is, for example, 5000, all brightness values ​​between 0 and 5000 are equally considered "dark" or "black." Typically, the brightness values ​​immediately above the cutoff value of 30 have a high information content. Here, for example, 10 brightness values ​​(e.g., 5001 to 5010) can be represented by one bit. For this reason, corresponding sampling during noise reduction in this range occurs at significantly shorter intervals.This is also shown, for example, in the left area of ​​Fig. 3.

[0115] Much higher brightness values, like very low brightness values, typically have a much lower information value. For example, R. 417530

[0116] - 14 -

[0117] Brightness values ​​from 60,000 to 65,536 can also be mapped to one bit. In this case, 5,536 brightness values ​​would be mapped to one bit. A lower sampling rate is sufficient for noise reduction in this scenario, as can be seen in the right-hand section of Figure 3.

[0118] This deviation in the mapping function is called non-equidistant quantization, whereby the numerical values ​​used here are only to be understood as examples.

Claims

R. 417530 - 15 - Claims 1. Method for processing raw sensor data (3) recorded in a vehicle during a test drive for obtaining validation data for validating driver assistance systems (4) with at least one sensor (2), comprising at least the following steps: A) Receiving the raw sensor data (3) from at least one sensor (2); B) Reducing a noise component from the raw sensor data (3) to generate cleaned sensor data (19); and C) Providing the cleaned sensor data (19) for storage.

2. The method of claim 1, wherein after step B) and before step C) the following process steps are carried out: c1) Generating synthesized sensor data (20) from the cleaned sensor data (19) by adding a synthetic noise component (21), c2) Performing a comparison of synthesized sensor data (20) with the raw sensor data (3) received in step A), checking whether any deviation between the synthesized sensor data (20) and the raw sensor data (3) received in step a) is within a specified tolerance range, where the provision of the corrected sensor data (19) in step C) only takes place if it has been determined in step c2) that the deviation is within the specified tolerance range.

3. Method according to claim 2, wherein in step c2) only synthesized sensor data (20) are generated from a subset of the cleaned sensor data and the comparison in step c2) is only made between the synthetic sensor data (20) and a corresponding part of the raw sensor data (3).

4. A method according to any of the preceding claims, wherein the at least one sensor (2) from which the sensor raw data (3) are received in step A) is at least one image, radar, lidar or ultrasonic sensor, in particular a camera, and the sensor raw data (3) are image data, in particular camera data. R. 417530 - 16 - 5. Method according to one of the preceding claims, wherein in step C) a lossless reversible compression (8) of the cleaned sensor data is carried out to prepare for storage of the cleaned sensor data (19), so that compressed sensor data (11) are produced for storage.

6. Method according to any of the preceding claims, wherein the noise reduction (7) is performed individually for pixels or groups of pixels.

7. Method according to one of the preceding claims, wherein in step B) noise components from the sensor raw data (3) are reduced which are caused by properties of the at least one sensor (2) from which the sensor raw data (3) are received in step A).

8. Method according to any of the preceding claims, wherein reducing the noise component in step B) comprises non-equidistant quantization of the sensor raw data (3).

9. Method according to one of the preceding claims, wherein at least one look-up table is used to reduce the noise component in step B).

10. Method according to one of the preceding claims, prior to reducing the noise component in step B) from several images in the sensor raw data (3) information is summarized to represent a non-linear behavior of a sensor.

11. Method according to one of the preceding claims, wherein the following step is carried out in addition to step C): D) Storing cleaned sensor data in a data storage system (12) for later provision as validation data for the validation of driver assistance systems (4).R. 417530 - 17 - 12. Method according to claim 11, wherein, in parallel to step A), further data (22) are received which were recorded in the vehicle and which are stored in the data storage system (12) in step D) together with the cleaned sensor data (19) in order to be available together with the cleaned sensor data (19) as validation data (23) for the validation of driver assistance systems (4).

13. Method according to one of claims 11 or 12, wherein the following step is carried out before step D): d1) Transmission of the cleaned sensor data(19) and / or the validation data (23) via a transmission interface (13) from the vehicle with the at least one sensor (2) from which the raw sensor data (3) were determined in step a) to a stationary data storage system (12).

14. Method according to one of the preceding claims, wherein meta-information contained in the raw sensor data (3) is not affected by the reduction of the noise component in step B) and is available for further processing in the cleaned sensor data (19).

15. Data processing system in a vehicle which is equipped for carrying out the method according to one of the preceding claims.

16. Vehicle equipped for conducting test drives to collect data for the creation of validation data (23) comprising a data processing system according to claim 15.

Images