Method for providing a loss value for training a second machine learning model using a first machine learning model

Abstract

The invention relates to a method for providing a loss value for training a second machine learning model using a first machine learning model. The invention further relates to a computer program, a device, and a storage medium for this purpose.

Description

[0001] R.416849

[0002] - 1 -

[0003] Description

[0004] title

[0005] Method for providing a loss value for training a second machine learning model using a first machine learning model

[0006] The invention relates to a method for providing a loss value for training a second machine learning model using a first machine learning model. The invention further relates to a computer program, a device, and a storage medium for this purpose.

[0007] State of the art

[0008] Driver assistance systems and automated driving require a good representation of the environment. Various modalities such as cameras, radar, lidar, and ultrasound can be used for this purpose.

[0009] Modern machine learning methods can be trained under supervision, meaning they use a large, labeled dataset. However, creating such a dataset is expensive and time-consuming.

[0010] Foundation models, such as CLIP, were trained on massive datasets and demonstrate particularly good performance across various tasks, as well as their ability to generalize. Typically, foundation models work with image and text data.

[0011] One challenge is the different dimensions in which the various modalities measure: images are represented in the image plane (pixels with coordinates in the image plane), lidar point clouds, for example, have R.416849

[0012] - 2 -

[0013] 3D coordinates (x, y, z) therefore also include the distance to the sensor. One challenge now is to associate the measured points of the point clouds with the corresponding pixels of the images (depth alignment).

[0014] In the document “Najibi et al. “Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Driving”, 2023, https: / / ar-xiv.org / abs / 2309.14491”, a multimodal autolabeling pipeline is developed. A pre-trained Vision Language Model (VLM) and an object detection network operating on lidar point clouds are used. The data consists of sequences of images and the corresponding lidar measurements. For each sequence, 3D bounding boxes are estimated, and the detected objects are tracked over a certain period. This makes it possible to associate the lidar points with the corresponding pixels from the measured image. This allows feature embeddings from the VLM to be assigned to the lidar points. This makes it possible to train a Lidar 3D object detection network, which also includes point-by-point feature embeddings.This allows the knowledge from the VLM to be transferred to the lidar network (knowledge distillation).

[0015] Disclosure of the invention

[0016] The invention relates to a method with the features of claim 1, a computer program with the features of claim 9, a device with the features of claim 10, and a computer-readable storage medium with the features of claim 11. Further features and details of the invention will become apparent from the respective dependent claims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program, the device, and the computer-readable storage medium according to the invention, and vice versa, so that a reciprocal reference is always possible with regard to the disclosure of the invention. R.416849

[0017] - 3 -

[0018] The invention relates in particular to a method for providing a loss value for training a second machine learning model using a first machine learning model, comprising:

[0019] Providing image data, wherein the image data comprises individual images and results from the acquisition of at least one camera sensor; providing sensor data, wherein the sensor data comprises individual samples, for example, respective measurements over a defined period of time, and results from the acquisition of at least one sensor, wherein the at least one sensor is configured for distance measurement, wherein the sensor data can be radar data, lidar data and / or ultrasonic data, and the at least one sensor can be configured accordingly as a radar, lidar and / or ultrasonic sensor.

[0020] Determining feature vectors for defined areas, for example patches of defined size or pixels, of a respective individual image of the image data using the first machine learning model, wherein the first machine learning model is preferably a foundation model.

[0021] Projecting a sample of the sensor data into a bird's eye view (BEV),

[0022] Providing the respective sample of sensor data and optionally also the bird's-eye view as input for the second machine learning model in order to obtain a feature map as output, in particular calculating a feature vector for each cell of the bird's-eye view.

[0023] Providing the loss value for training the second machine learning model, wherein the loss value for the feature map of the second machine learning model is calculated based on the bird's-eye view representation and the feature vectors determined using the first machine learning model, wherein the loss value is, in particular, a similarity between the feature vectors determined using the first machine learning model and the feature vectors calculated using the second machine learning model for each cell of the bird's-eye view representation, wherein the similarity between the feature vectors can be calculated, for example, by a cosine similarity or a mean squared error. R.416849

[0024] - 4 -

[0025] The image data and the sensor data represent, in particular, overlapping areas of the same scene. The at least one camera sensor and the at least one sensor designed for distance measurement can, for example, be arranged on a vehicle and, starting from the vehicle, capture its surroundings. The sensors are preferably oriented in the same perspective, i.e., they preferably point in the same direction.

[0026] The following steps may be necessary when projecting the respective sensor data sample into the bird's eye view (BEV). If the sensor data is in a polar coordinate system, it must be converted to a Cartesian coordinate system:

[0027] x = r • cos(0)

[0028] y = r • sin(0)

[0029] where r is, in particular, the distance of an object from the sensor and 0 is an angle relative to the sensor axis. If the sensor also measures the elevation angle, the projection formulas can be extended accordingly.

[0030] Subsequently, the position and orientation of the sensor relative to the desired perspective (e.g., vehicle coordinate system or world coordinate system) can be determined. This is done, for example, by matrix multiplication using a transformation T. sensor ^ worid This is achieved. Since the bird's-eye view is a two-dimensional representation, the sensor data is preferably projected onto a plane along a Z-axis (height). The Cartesian coordinate system can then be converted into a grid or image format, and each point can be divided into a grid cell. The cell size depends primarily on the desired resolution.

[0031] By using the first machine learning model and the provided loss value, the training of the second machine learning model can be accelerated, thus increasing efficiency and reducing computational requirements. R.416849

[0032] - 5 -

[0033] A further advantage can be achieved within the scope of the invention if the method further comprises:

[0034] Using the provided loss value during the training of the second machine learning model, minimizing the loss value during training.

[0035] This ensures that the second machine learning model can be optimized by minimizing the loss value and that it can learn to predict feature vectors based on sensor data.

[0036] According to a further advantage, the procedure may also include:

[0037] Fine-tuning the trained second machine learning model for classification and / or object detection and / or semantic segmentation within the framework of at least partially automated driving.

[0038] This ensures, in particular, that the trained second machine learning model is able to classify and detect objects and / or semantically segment an environment, for example other vehicles or pedestrians, which can be relevant for functions within the framework of at least partially automated driving, such as automatic distance control or even fully automated driving.

[0039] Optionally, the procedure may also include:

[0040] Determining an initial mask containing all non-empty cells of the bird's-eye view representation,

[0041] Masking out all occupied cells of the first mask that are at least partially obscured from the perspective of at least one sensor by at least one occupied cell, in order to obtain a second mask.

[0042] Masking means, in particular, that these cells are replaced by empty cells. For example, cells that are at least 50% obscured can be masked out. This allows a focus to be achieved on the relevant areas within the bird's-eye view, since the obscured cells are not expected to be represented in the image data. This leads, in particular, to improved accuracy in training the second machine learning model.

[0043] - 6 -

[0044] Furthermore, the invention may provide that the method further comprises:

[0045] Extrapolating the bird's-eye view representation, especially the second mask, to a resolution of the feature map, particularly to obtain a third mask.

[0046] This allows the accuracy of the training of the second machine learning model to be further improved, as the extrapolation can provide a resolution of the bird's-eye view adapted to the feature map.

[0047] The procedure may also include:

[0048] Projecting a center point of a respective cell of the bird's-eye view representation, in particular the third mask, into an image plane, in particular using intrinsic and / or extrinsic calibration matrices of the at least one sensor and / or the at least one camera sensor,

[0049] Determine one of the identified feature vectors in whose defined area the projected center point of the respective cell lies, in order to assign the respective identified feature vector to the respective cell of the bird's-eye view representation.

[0050] The procedure may also include:

[0051] Using the trained second machine learning model for object detection based on text input (English: "Open Vocabulary Object Detection"),

[0052] The following steps are planned as part of object detection based on text input:

[0053] Calculating a feature vector for each text input using the first machine learning model,

[0054] Calculating feature vectors for each cell in the bird's-eye view using the second machine learning model, determining a similarity between the feature vector of the respective text input and each feature vector in the bird's-eye view, e.g. based on a cosine similarity or a mean squared error, comparing the determined similarity with a defined threshold, R.416849

[0055] - 7 -

[0056] Output an object detection at the corresponding cell in the bird's-eye view if the determined similarity exceeds the defined threshold.

[0057] Furthermore, the procedure may also include:

[0058] Generating an image for each feature vector in each cell of the bird's-eye view representation using the trained second machine learning model to visualize the sensor data and / or provide synthetic sensor data through the generated images.

[0059] It is possible that the method according to the invention is used in a vehicle. The vehicle can be, for example, a motor vehicle and / or a passenger vehicle and / or at least partially automated / autonomous. The vehicle can have vehicle equipment, for example, for providing an autonomous driving function and / or a driver assistance system. The vehicle equipment can be designed to control the vehicle at least partially automatically and / or accelerate and / or brake and / or steer.

[0060] The machine learning system, particularly in the form of the second machine learning model, is trained, or fine-tuned, specifically for classification and object detection. This fine-tuning can involve training the machine learning system or model using a training dataset for classification and / or object detection based on sensor data, such as point clouds or digital image data, especially pixel attributes. The sensor data or digital images can, for example, result from a recording by at least one sensor, such as a radar, lidar, and / or ultrasonic sensor, preferably from a vehicle and particularly preferably from the vehicle's surroundings during a journey. The recording is possible, for example, by at least one radar, lidar, and / or ultrasonic sensor of the vehicle.The classification can be used to detect objects in an environment depicted by sensor data or digital images and / or to capture a traffic scene. R.416849.

[0061] - 8 -

[0062] The classification can be used for various technical applications. One example is its use in vehicles. Based on the classification, and in particular at least one classification result, at least one control action, preferably for a vehicle or another technical system, can be initiated and / or carried out.

[0063] A classification result may include at least one of the following results and / or be specific to at least one of the following: a category of objects, an identification of objects, a position of objects and / or obstacles (e.g., in the direction of travel or beside the direction of travel), a presence of obstacles, a description of a traffic scene, a hazard warning, a number of objects, a type and / or position of road markings and / or a road boundary, a position and / or condition of traffic signal equipment, a position of a roadway, or the like.

[0064] Based on the classification result, at least one control action can be initiated and / or carried out for the vehicle. The control action can include at least one of the following: braking, steering, accelerating, overtaking, emergency braking, activating an alarm system, activating hazard warning lights, activating a turn signal, controlling the lights, or the like.

[0065] Classification allows, for example, the detection of an obstacle, regardless of whether it is directly in the direction of travel or to the side. Depending on the location (e.g., based on the expected vehicle trajectory), an appropriate control action, such as braking or swerving, can be initiated.

[0066] For example, braking can also be initiated if the classification indicates that there are obstacles in the direction of travel and / or a collision is likely. It is also conceivable that a lane and / or lane boundary could be detected based on the classification in order to move the vehicle, at least partially automatically, along the lane through the control action. R.416849

[0067] - 9 -

[0068] Classification can also encompass object detection. This refers specifically to classifying whether or not objects are present in certain areas of the environment. Furthermore, the term "classification" can also refer to semantic segmentation, particularly in the form of pixel-level classification.

[0069] Accordingly, the training can result in at least one trained, or finely tuned, machine learning model that can be used for classification and / or object detection. Its use, and thus the inference, can be implemented, for example, in a vehicle. The input data points can be, for example, point clouds or pixels from image data, or be based on them, in order to perform the classification and / or object detection of the data points based on the point clouds or pixels. The input data can include sensor and / or image data, which at least partially results from acquisition with a sensor, preferably a radar, lidar, or ultrasonic sensor, and / or which has been at least partially synthesized, i.e., in particular, replicates the real data of a sensor.Specifically, the system can be designed so that the values ​​of points, preferably point cloud data or pixels, from the sensor data represent the environment of a sensor and / or a vehicle and / or a traffic scene. Classification and / or object detection based on these values ​​can be implemented. This allows, for example, the detection of objects within the traffic scene. The sensor data or image data can be, for example, images from a radar sensor and / or an ultrasonic sensor and / or a LiDAR sensor and / or a thermal imaging camera. Accordingly, the images can also be presented as radar images and / or ultrasonic images and / or thermal images and / or LiDAR images.

[0070] Also part of the invention is a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by at least one computer, cause it to execute the method according to the invention. Thus, the computer program according to the invention offers the same advantages as those described in detail with reference to a method according to the invention. R.416849

[0071] - 10 -

[0072] The invention also relates to a data processing device configured to execute the method according to the invention. The device can, for example, comprise at least one computer which executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data storage device can also be provided in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0073] The invention may also relate to a computer-readable storage medium which contains the computer program according to the invention and / or includes instructions which, when executed by at least one computer, cause it to execute the method according to the invention. The storage medium is, for example, designed as a data storage device such as a hard drive and / or non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0074] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or carried out automatically.

[0075] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination. The drawings show:

[0076] Fig. 1 is a schematic visualization of a method, a technical system, a device, a storage medium and a computer program according to embodiments of the invention, R.416849

[0077] - 11 -

[0078] Fig. 2 shows a schematic representation of cells of a bird's-eye view view, a camera sensor and a sensor according to exemplary embodiments of the invention.

[0079] Fig. 1 schematically shows a method 100, a technical system 1, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.

[0080] Fig. 1 shows, in particular, an embodiment of a method 100 for providing a loss value for training a second machine learning model using a first machine learning model. In a first step 101, image data is provided, wherein the image data comprises individual images and results from the acquisition of at least one camera sensor 3a. In a second step 102, sensor data is provided, wherein the sensor data comprises individual samples and results from the acquisition of at least one sensor 3b, the at least one sensor 3b being configured for distance measurement. In a third step 103, feature vectors for defined areas of each individual image of the image data are determined using the first machine learning model. In a fourth step 104, each sample of the sensor data is projected into a bird's-eye view.In a fifth step (105), the respective samples of sensor data and, optionally, the bird's-eye view are provided as input for the second machine learning model to obtain a feature map as output. In a sixth step (106), the loss value for training the second machine learning model is provided, whereby the loss value for the feature map of the second machine learning model is calculated based on the bird's-eye view and the feature vectors determined using the first machine learning model.

[0081] According to exemplary embodiments, the invention makes it possible to transfer features of a first machine learning model (teacher model), in particular a neural network that works on the basis of image data, to a second machine learning model, in particular a neural network, which, for example, works on radar, lidar, and / or ultrasound data. C, studentR.416849

[0082] - 12 -

[0083] model”). This corresponds in particular to a knowledge distillation from a vision model to a model of another modality. The first machine learning model (“teacher model”) can in particular be a foundation model.

[0084] The features of the first machine learning model can be used as a training signal, or to provide a loss value, within the framework of training the second machine learning model, whereby the second machine learning model works in particular on a bird's eye view (BEV).

[0085] For each cell in the bird's-eye view, corresponding areas in a respective captured image are preferably determined. Feature embeddings, or feature vectors, of these image areas can be used as a training signal for the respective cell in the bird's-eye view.

[0086] In contrast to the state of the art, feature embeddings, or feature vectors, are determined not for each measured point of a point cloud, but for each cell in the bird's-eye view representation.

[0087] The captured images, in particular, lack depth information. One challenge, therefore, can be to project the images appropriately onto the bird's-eye view (depth alignment) or to associate the cells of the bird's-eye view with the image areas.

[0088] According to exemplary embodiments of the invention, direct distance measurements from sensor 3b (e.g., radar, lidar, and / or ultrasound) are used to associate the cells of the bird's-eye view with the image areas. In contrast to the prior art, this does not require depth estimation of the image data (depth from mono) or a stereo camera that provides depth information. In contrast to the prior art, a multimodal autolabeling pipeline is also not necessary. R.416849

[0089] - 13 -

[0090] Radar sensors can generally measure objects that are behind another object. For example, they can measure a vehicle driving in front of another vehicle (an "ego vehicle") and a vehicle driving behind it. However, the corresponding image generally only shows a vehicle driving in front of it. According to embodiments of the invention, this is taken into account, and objects that are not visible in the image are preferably discarded accordingly.

[0091] The proposed invention offers the following advantages over the prior art, for example: No complex multimodal autolabeling pipeline is required. This makes the method according to the invention particularly easier to implement and saves computing resources. In particular, no object detection (bounding boxes) needs to be determined to calculate the training signal or provide the loss value. In particular, no sequences of measurements are required. The method according to the invention can operate with single pairs of image data and radar, lidar, and / or ultrasound data. Therefore, more datasets can be used, as datasets without sequences are also usable. In particular, no object tracks need to be calculated. This saves computing resources. The second machine learning model ("student model") is preferably based on feature embeddings, or...Feature vectors are trained with respect to cells of the bird's-eye view, not with respect to individual points. This is particularly more computationally efficient. The cell size determines the fine or coarse cell resolution. Radar sensors can measure objects that are obscured and not visible in the image. This can be taken into account according to the invention. The method according to the invention allows a general backbone of the radar, lidar, and / or ultrasound network, i.e., the second machine learning model, to be pre-trained. Therefore, the second machine learning model does not necessarily have to be specific to object detection, but can be used for various other tasks such as freespace segmentation.

[0092] Other state-of-the-art approaches require, in particular, depth information for the image data, such as through depth-from-mono algorithms or R.416849.

[0093] - 14 -

[0094] by using a stereo camera. According to the invention, this is particularly unnecessary.

[0095] In contrast to approaches that only consider one feature embedding vector per image, the method according to exemplary embodiments of the invention particularly considers one feature embedding vector per cell in the bird's-eye view representation.

[0096] This makes the method according to the invention more suitable for applications where the position of objects is important, such as object detection or freespace estimation.

[0097] The invention can be used in a technical system 1 such as, for example, in driver assistance systems of a vehicle or, for example, in an autonomously driving vehicle or a robot.

[0098] The invention can be used for the analysis of data acquired by a camera sensor 3a and a sensor 3b. The camera sensor 3a and the sensor 3b, respectively, can determine measured values ​​of the environment in the form of sensor signals, which may be provided, for example, by digital sensor data such as point clouds or digital images from, for example, camera, radar, LiDAR, ultrasonic sensors, motion images, or thermal images.

[0099] The invention can be used to classify sensor data, detect the presence of objects in the sensor data, or perform semantic segmentation of the sensor data, e.g., with regard to traffic signs, road surfaces, pedestrians, vehicles, and road users. This is done on the basis of low-level features (e.g., edges or pixel attributes in images).

[0100] The invention can be used to determine one or more continuous values, i.e., to perform a regression analysis, e.g., regarding a distance, velocity, acceleration, or tracking of an element, such as an object, in the data. This is done on the basis of low-level features (e.g., edges or pixel attributes in images). R.416849

[0101] - 15 -

[0102] The invention can be used to calculate a control signal for controlling a technical system 1, such as a computer-controlled machine, a robot system, a vehicle, a household appliance, or a system for transmitting information, such as a monitoring system or a medical (imaging) system. For this purpose, data (e.g., scalar time series), in particular from a sensor, namely radar, lidar, and / or ultrasonic sensor, are analyzed, and the technical system is operated accordingly.

[0103] The invention relates in particular to an upstream component in the machine learning toolchain. Specifically, the invention does not directly improve a machine learning system that can be used for the applications mentioned above, but rather is a method for training such a machine learning system (including learning a strategy). Once the machine learning system has been trained in this way, it can be used for the downstream applications as described above.

[0104] The invention is, in particular, a type of knowledge distillation. In this knowledge distillation, the large, high-performing first machine learning model (the "teacher model") is used. The output of the first machine learning model, or specific features of the first machine learning model, are preferably used as a supervision signal for the second machine learning model (the "student model"). The second machine learning model is, in particular, a smaller machine learning model, especially a neural network, which is optimized during training to predict the features of the first machine learning model. This enables high performance to be achieved with the second machine learning model.

[0105] The method according to the invention is described below by way of example using a radar sensor. However, it can also be used with other sensors that measure distance, such as lidar or ultrasonic sensors. One objective is therefore, in particular, to train a radar encoder, i.e., a Ma-R.416849

[0106] - 16 -

[0107] A machine learning model receives radar data as input and calculates a representation of that data. This radar encoder, also called the backbone, can then be used for various tasks. For example, by adding a small machine learning model, particularly a neural network (called the "head"), a machine learning model for object detection, freespace estimation, or other perception tasks can be created. This smaller, second machine learning model can then be advantageously fine-tuned for the specific task using a small amount of labeled data.

[0108] Preferably, pairs of image and radar data are used as sensor data, whereby, in particular, corresponding camera sensors 3a and sensors 3b measure overlapping areas of a scene. For example, camera sensor 3a and a radar sensor 3b can be mounted on a vehicle 1 to acquire such sensor data. While the vehicle 1 is moving, both sensors 3a and 3b can acquire sensor data simultaneously. The sensor data does not need to be labeled.

[0109] The first machine learning model, i.e., specifically the teacher vision model, is preferably a high-performance machine learning model that can determine or calculate a feature vector, especially an embedded feature vector, for areas in the image, e.g., square patches, or for each pixel in the image. This could be, for example, a model like OWL-ViT or OpenSeg.

[0110] The method according to exemplary embodiments of the invention comprises, for example, the following steps, which are carried out from the sensor data, in particular for each sample, i.e., pair of image and radar data:

[0111] Using the first machine learning model, in particular the teacher vision model, feature vectors are preferably calculated for each patch or pixel in the image.

[0112] The sensor data, i.e., in particular the radar data, e.g. radar point clouds, are then preferably displayed on a Birds Eye View (BEV) grid, or an R.416849

[0113] - 17 -

[0114] Bird's-eye view, projected. The cell resolution can be freely chosen, e.g. 0.5 meters x 0.5 meters.

[0115] Then, preferably, a first mask M1 is determined containing all non-empty cells of the bird's-eye view representation, or BEV Grid.

[0116] For all occupied cells, the following is preferably carried out: In particular, all occupied cells 2b from M1 are masked out if they are completely and / or partially obscured by another occupied cell 2a that is closer from the perspective of the camera sensor 3a or sensor 3b (see Fig. 2). The camera sensor 3a and the sensor 3b are preferably aligned perspectively, as shown in Fig. 2. The points in Fig. 2 represent, in particular, the sensor data, for example, points of a radar point cloud. That is, an obscured cell 2b is, in particular, at a greater distance than the cell 2a under consideration and must, in particular, lie completely or partially within the area defined by the corner points of the cell 2a under consideration and the position of the camera sensor 3a or sensor 3b (see Fig. 2). For example, cells that are at least 50% obscured can be masked out.This resulting second mask M2 is preferably stored. It therefore preferably only includes the occupied cells 2a that are not hidden.

[0117] The second machine learning model, i.e., in particular the radar encoder, preferably receives the radar data of the pair as input and outputs, in particular, a feature map.

[0118] The dimensions of the feature map are, for example, x, y, feature vector dimension. The mask M2 is then interpolated or extrapolated to the resolution (x, y). This allows a third mask M3 with the resolution (x, y) to be obtained.

[0119] Advantageously, the resolution (x,y) can be chosen to be lower than the resolution of mask M2. This allows the hidden cells to be calculated accurately without the second machine learning model, or the radar encoder, having to calculate a very high-resolution feature map. R.416849

[0120] - 18 -

[0121] For all occupied cells in M3, the following is performed, for example: A center point of each cell can be projected onto the image plane using intrinsic and extrinsic calibration matrices of the radar and camera sensors. Let the projected point be, for example, P. Subsequently, the feature vector of the first machine learning model, i.e., in particular the vision model, can be selected that belongs to the area (patch or pixel) in which point P lies. This is stored, for example, in a list L.

[0122] Now, for the feature vectors, or feature map, of the second machine learning model, or radar encoder, a loss value can be calculated or minimized for all occupied cells M3 and all corresponding feature vectors from the list L. The loss value can be, for example, the mean square error or the cosine similarity loss. This loss value can then be used to train the second machine learning model, or radar encoder.

[0123] The method according to the invention can be implemented with little additional computational effort compared to a typical training of a machine learning model or neural network.

[0124] The following extensions of the method according to the invention are possible: Instead of just one feature map, several feature maps of the second machine learning model, or radar encoder, can be used. The feature maps can have different resolutions, and the corresponding steps described above can be performed for each feature map. The total loss value is obtained, for example, as the sum or average of the individual loss values.

[0125] Furthermore, within the scope of the present invention, several feature maps of the second machine learning model, or radar encoder, can be used as follows. The multiple feature maps can be processed together with additional layers of a machine learning model, in particular a neural network, such as linear layers with activation functions, deconvolutions, or convolutions, to form a common feature map. This new resulting R.416849

[0126] - 19 -

[0127] The feature map can then be used in the corresponding steps described above.

[0128] The method according to the invention can be extended to multiple video and radar, lidar, and / or ultrasonic sensors. For this purpose, the sensor data, for example point clouds, from the radar, lidar, and / or ultrasonic sensors can be aggregated in a common coordinate system of the bird's-eye view before the first step. If, for example, multiple cameras are used, the corresponding step described above can be performed for each camera image. With this extension, it is possible, for example, to train a backbone that processes sensor data from a 360° view around a vehicle.

[0129] Another application is open vocabulary object detection: The trained second machine learning model, i.e., in particular the trained radar, lidar, and / or ultrasound encoder, can also be used for open vocabulary object detection. For example, a text input (prompt) such as "there is a car," "car," or "there is a pedestrian" is used. Using a text encoder of the first machine learning model, i.e., preferably the base model, a feature vector P can be calculated for the text input. Using the second machine learning model, i.e., in particular the radar, lidar, and / or ultrasound encoder, feature vectors for each cell in the bird's-eye view can then be calculated. The feature vector P of the text encoder is then preferably compared with each feature vector in the bird's-eye view, e.g., by calculating the cosine similarity. After optional normalization of the values, e.g.,Using a softmax function, the values ​​are preferably compared to a defined threshold. For each value above the threshold, an object detection can be output at the corresponding cell in the bird's-eye view. This makes it possible, for example, to calculate detections for any type of object that can be described by text.

[0130] Furthermore, the sensor data can be visualized and synthetic sensor data can be generated: A generative machine learning model can be used for this purpose (Wer-R.416849).

[0131] - 20 -

[0132] This model can generate images based on the features of the first machine learning model, or vision foundation model. Then, an image can be generated for each feature vector in each cell of the bird's-eye view. This makes it possible to visualize sensor data from radar, lidar, and / or ultrasound sensors, or to generate synthetic image data based on radar, lidar, and / or ultrasound data.

[0133] The preceding explanation of the embodiments describes the present invention solely by way of examples. Naturally, individual features of the embodiments can be freely combined with one another, provided this is technically feasible, without departing from the scope of the present invention.

Claims

R.416849 - 21 - Claims 1. Method (100) for providing a loss value for training a second machine learning model using a first machine learning model, comprising: Providing (101) image data, wherein the image data comprise individual images and result from the acquisition of at least one camera sensor (3a), Providing (102) sensor data, wherein the sensor data comprise individual samples and result from the acquisition of at least one sensor (3b), wherein the at least one sensor (3b) is configured for distance measurement, Determining (103) feature vectors for defined areas of each individual image of the image data using the first machine learning model, Projecting (104) a respective sample of the sensor data into a bird's-eye view representation, Providing (105) the respective sample of sensor data as input for the second machine learning model in order to obtain a feature map as output, Providing (106) the loss value for training the second machine learning model, wherein the loss value for the feature map of the second machine learning model is calculated based on the bird's-eye view representation and the feature vectors determined using the first machine learning model. R.416849 - 22 - 2. Method (100) according to claim 1 , characterized by that the procedure (100) further includes: Using the provided loss value during the training of the second machine learning model, minimizing the loss value during training.

3. Method (100) according to claim 2, characterized by that the procedure (100) further includes: Fine-tuning the trained second machine learning model for classification and / or object detection and / or semantic segmentation within the framework of at least partially automated driving.

4. Method (100) according to one of the preceding claims, characterized in that that the procedure (100) further includes: Determining an initial mask containing all non-empty cells of the bird's-eye view representation, Masking out all occupied cells of the first mask which, from a perspective of the at least one sensor (3b), are at least partially obscured by at least one respective occupied cell, in order to obtain a second mask.

5. Method (100) according to any one of the preceding claims, characterized in that that the procedure (100) further includes: Extrapolating the bird's-eye view representation, especially the second mask, to a resolution of the feature map, particularly to obtain a third mask. R.416849 - 23 - 6. Method (100) according to one of the preceding claims, characterized in that that the procedure further includes: Projecting the center point of each cell of the bird's-eye view representation, especially the third mask, onto an image plane, Determine one of the identified feature vectors in whose defined area the projected center point of the respective cell lies, in order to assign the respective identified feature vector to the respective cell of the bird's-eye view representation.

7. Method (100) according to any one of claims 2 to 6, characterized by that the procedure (100) further includes: Using the trained second machine learning model for object detection based on text input, The following steps are planned as part of object detection based on text input: Calculating a feature vector for each text input using the first machine learning model; calculating feature vectors for each cell in the bird's-eye view using the second machine learning model. Determining a similarity between the feature vector corresponding to the respective text input and each feature vector in the bird's-eye view representation, Comparing the determined similarity with a defined threshold, Output an object detection at the corresponding cell in the bird's-eye view if the determined similarity exceeds the defined threshold. R.416849 - 24 - 8. Method (100) according to any one of claims 2 to 7, characterized by that the procedure (100) further includes: Generating an image for each feature vector in each cell of the bird's-eye view representation using the trained second machine learning model to visualize the sensor data and / or provide synthetic sensor data through the generated images.

9. Computer program (20), comprising instructions which, when the computer program (20) is executed by at least one computer (10), cause it to execute the method (100) according to one of the preceding claims.

10. Device (10) for data processing, which is configured to carry out the method (100) according to any one of claims 1 to 8.

11. Computer-readable storage medium (15) comprising instructions which, when executed by at least one computer (10), cause it to perform the steps of the method (100) according to any one of claims 1 to 8.

Images