Determining a scaled depth image with monocular image capture and on the basis of rail path data

The method for determining a scaled depth image using monocular image acquisition and rail path data addresses the challenges of complex railway environments by employing AI-based depth estimation and track models to generate accurate depth maps, enhancing object detection and reducing computational complexity.

WO2026233588A1PCT designated stage Publication Date: 2026-11-12SIEMENS MOBILITY GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/055135
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-05
Filing Date
2026-02-25
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Rail vehicles face challenges in accurately perceiving their environment due to complex railway conditions, varying visibility, and the need for rapid obstacle detection, which is computationally intensive when using multiple sensors like LiDAR and cameras, and monocular camera systems require complex algorithms for depth estimation.

Method used

A method using monocular image acquisition combined with rail path data to determine a scaled depth image, employing AI-based depth estimation and a track model to generate metrically accurate depth maps, eliminating the need for complex computer vision methods and reducing computational intensity.

Benefits of technology

Enables efficient, resource-effective 3D environment perception for rail vehicles by using visible rails as a scale, ensuring reliable object detection and classification, and reducing computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026055135_12112026_PF_FP_ABST
    Figure EP2026055135_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for determining a scaled depth image with monocular image capture combined with rail path data. In the method, monocular image data (BD) are automatedly captured from an environment (U) of a rail vehicle (2) by a monocular image recording unit (3) carried by the rail vehicle (2). Furthermore, an unscaled depth map (TKU) is determined by an AI-based depth estimation method (TSV). A rail section model (M) of the rail section (4) currently being traveled on by the rail vehicle (2) is also generated with an absolute scale (AS). The rail section model (M) is then projected into the unscaled depth map (TKU). Finally, a scaled depth map (TKS) is generated on the basis of the unscaled depth map (TKU) and on the basis of the absolute scale (AS) of the rail section model (M) projected into the unscaled depth map (TKU). The invention also relates to a method for object detection. The invention also relates to a method for training an AI-based depth estimation method. The invention furthermore relates to a depth data determining device (40). The invention moreover relates to an object detection device (50). The invention additionally relates to a rail vehicle (2).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] 202505294

[0002] 1

[0003] Description

[0004] Determining a scaled depth image using monocular image acquisition and based on rail path data

[0005] The invention relates to a method for determining a scaled depth image using monocular image acquisition combined with track path data. Furthermore, the invention relates to a method for object detection. The invention also relates to a depth data acquisition device. Additionally, the invention relates to an object detection device. Finally, the invention relates to a rail vehicle equipped with such an object detection device.

[0006] Implementing 3D perception for rail vehicles presents several technical challenges related to the accuracy of capturing and interpreting the environment in front of the vehicle. A key problem is the complexity of the railway environment, which can include varying terrain, weather conditions, and low temperatures. Visibility also varies, making real-time perception difficult. Since rail vehicles often travel at high speeds, the system must quickly detect objects in the vicinity of the track and react immediately if necessary. A rail vehicle should react to obstacles or changes as far in advance as possible, which requires long-range sensors and rapid data processing. The fusion of data from multiple sensors, such as LiDAR, cameras, and radar, is essential for a complete 3D understanding of objects and track conditions.However, this approach is extremely computationally intensive.

[0007] Another challenge lies in ensuring reliable detection and classification of objects such as animals, people, or inanimate objects, even in complex or ambiguous environments. Furthermore, the presence of tunnels, bridges, and other infrastructure elements can impair sensor performance and complicate the process. These challenges necessitate the application of sophisticated algorithms, advanced sensors, and real-time processing capabilities, as well as the ability to guarantee reliable 3D perception for rail vehicles.

[0008] The publications Badrloo, Varshosaz, Pirasteh and Li: Image-Based Obstacle Detection Methods for the Safe Navigation of Unmanned Vehicles: A Review, 2022 and Ristic-Dur-rant, Franke and Michels: A Review of Vision-Based On-Board Obstacle Detection and202505294

[0009] 2

[0010] Distance Estimation in Railways, 2021 provides a comprehensive overview of approaches to environmental perception for railway vehicles. A wide range of sensors can be used to capture the 3D environment. Depending on the application and requirements, there are many possible combinations of electro-optical sensors (cameras, LiDAR), electromagnetic sensors (radar), and other sensors and algorithms for generating 3D information. Spatial resolution, accuracy, and measurement range depend on the sensor capabilities, and a skillful combination of complementary sensors and the use of intelligent signal processing techniques are necessary for efficient environmental perception of a railway vehicle.However, the performance of these sensor systems may be limited due to various circumstances (daylight requirements, eye safety aspects, limited resolution or limited measuring range), as well as due to limitations in energy consumption, handling complexity and cost.

[0011] Comparatively inexpensive and less complex monocular camera technologies can offer an alternative to elaborate stereo sensor systems. However, the reduced sensor complexity usually leads to more complex and complicated algorithms for processing the captured image data. For example, "Structure from Motion" or "Optical Flow" is used to obtain scale-free distance measurements. Absolute distances can be determined using model assumptions about the railway lines and known track gauges. The recorded rails are either viewed from a first-person perspective and used as a kind of ruler, or they are depicted from a so-called bird's-eye view (using an inverse perspective transformation). This generates (sparse) depth representations to serve as a basis for approaches to perceiving the 3D environment.

[0012] In the last decade, monocular depth estimation (abbreviated MDE) has gained considerable attention. MDE is explained in detail in Jin-woo Bae, Kyumin Hwang, and Sunghoon Im: A Study on the Generality of Neural Network Structures for Monocular Depth Estimation, 2024. MDE is a technique for estimating the distance between an object and a camera and uses only single RGB images (RGB stands for Red, Green, Blue) or even single monochromatic images. MDE bridges the gap between LiDAR sensors and RGB cameras. MDE is compatible with structure-from-motion methods (SFM, a method for obtaining 3D information by superimposing time-shifted images).

[0013] 3

[0014] Related to this are methods that derive depth information from feature matches in image sequences. This technique plays a crucial role in self-supervised training or training of MDE neural networks using labeled training data. However, unlike SFM, MDE depth estimation is performed using only a single image, based on a trained neural network. The continuous development, dissemination, and interest in self-supervised deep learning methods (deep learning is a type of artificial intelligence or machine learning based on multilayer neural networks) have increased. Simultaneously, the performance of monocular depth estimation with MDE has improved significantly, both in terms of network efficiency and generalization.Therefore, the application of MDE in autonomous driving (abbreviated as “AD”) can be considered.

[0015] A crucial aspect of MDE is determining the absolute scale in the generated depth maps. Generally, no information about the metric depth can be derived from a single RGB or monochromatic image alone. However, several methods exist that can be employed, either during training or after applying an artificial neural network, to determine the scale. In the automotive field, understanding the geometric relationship between the camera and the ground (with prior knowledge of the camera's absolute height) is an effective approach to address this problem. Another approach involves recognizing objects of known size within the image and applying the scale information to the entire depth map.

[0016] Furthermore, the scale can be taken into account during training, and the network learns the scale. This means that depth prediction is directly linked to the extrinsic and intrinsic properties of the camera as parameters. However, any subsequent change to these parameters would require retraining the network.

[0017] The task is therefore to provide a method and a device for depth determination in monocularly acquired image data.

[0018] This task is accomplished by a method for determining a scaled depth image using monocular image acquisition combined with rail path data according to claim 1,202505294

[0019] 4

[0020] a method for training an AI-based depth estimation method according to claim 9, a method for object detection according to claim 10, a depth data determination device according to claim 11, an object detection device according to claim 12 and a rail vehicle according to claim 13 solved.

[0021] In the inventive method for determining a scaled depth image using monocular image acquisition combined with track path data, image data from the surroundings of a rail vehicle are automatically acquired by a monocular image acquisition unit carried by the rail vehicle. As will be explained below, the track path data preferably comprises a model-based description of one or more track paths, which is adapted to the acquired image data. A monocular image acquisition unit is understood to be a passive sensor unit for acquiring image data that captures image data from only a single perspective at a given time. The monocular image acquisition unit is thus distinguished in particular from a stereo image acquisition unit, which simultaneously captures image data from multiple or at least two perspectives.

[0022] Based on the acquired monocular image data, an unscaled depth map is determined using an AI-based depth estimation method (AI being an abbreviation for "artificial intelligence" and encompassing methods based on artificial neural networks), preferably an MDE method. A depth map (or depth image) is an image or image channel containing information relating to the distance of the surfaces of scene objects. Each pixel of the image represents the distance to the camera. Unscaled depth maps accurately represent the relative sizes of the objects and their distances from one another in the image, but do not contain accurate metric information.

[0023] In parallel, a track model of the railway line currently being traveled by the rail vehicle is generated using an absolute scale. An absolute scale is defined as a scale that specifies the distances of points on the track to the image capture unit or camera, and, if applicable, also the distances between points on the track itself, as absolute values.

[0024] This railway track model is then projected onto the unscaled depth map using the known intrinsic camera parameters. This allows pixels of the unscaled map to be directly selected.

[0025] 5

[0026] The depth map is assigned a metric value. The necessary "matching" or registration of a rail model with the monocular image data or the unscaled depth map during projection is preferably carried out according to a method described in European patent application EP 4253 188 A1.

[0027] Finally, a scaled depth map is generated based on the unscaled depth map and on the absolute scale of the track model projected onto the unscaled depth map. Since the distance of individual points on the tracks to the image acquisition unit is known, further points can easily be determined by interpolation or extrapolation, or by another mathematical adaptation or regression method.

[0028] The method according to the invention enables efficient and robust 3D measurement of the area in front of a rail vehicle. Compared to lidar or stereo camera systems, this method offers the following advantages:

[0029] The key difference to monocular camera systems already used in autonomous driving is the use of visible rails as a scale for generating metrically accurate depth images. The rails are always visible, thus eliminating the need for complex computer vision methods to determine the ground plane or other reference objects in the environment.

[0030] The method according to the invention can be used in online and offline scenarios: possible scenarios include monitoring the track area during autonomous driving, event-based recording, and offline processing of the recorded data for a visual representation of the monitored area. Preferably, only one camera is used as the image acquisition unit, thereby avoiding complex stereo calibration and the dependence thereon.

[0031] The proposed method can be implemented in a much more resource-efficient and cost-effective manner compared to the use of lidar-based systems or stereo cameras.

[0032] The idea underlying the invention is to provide a method and system for 3D environment perception that incorporates a monocular image acquisition unit, preferably a monocular RGB camera or a monocular monochromatic camera.

[0033] 6

[0034] The camera incorporates an MDE approach along with track information to generate a metric image with a correct depth image and accurate depth data.

[0035] In the inventive method for training an AI-based depth estimation method, which is to be used within the framework of the inventive method for determining a scaled depth image with monocular image acquisition combined with rail path data, monocular training image data labeled with a depth estimate are first received. This data comprises rails traversed by a rail vehicle and their surroundings. Furthermore, relative depth information or relative depth data are determined based on the monocular training image data using the AI-based depth estimation method to be trained during a training process.

[0036] As part of the adaptation of a KL-based model (KL stands for "artificial intelligence" and includes in particular models based on artificial neural networks), relative depth data are determined as result data based on the KL-based model to be adapted.

[0037] Furthermore, an adapted AI-based depth estimation method is generated based on the depth information or depth data determined as result data and on the basis of the depth estimation of the respective label of the training image data.

[0038] Since the correct relative depth data are known in advance, the AI-based model is now adapted based on the depth data estimated by the AI-based model and on the basis of the correct relative depth data.

[0039] The two steps of depth estimation and adaptation are repeated until a predetermined quality criterion is met for the depth estimation performed, for example, a maximum permissible deviation from the previously known depth data is reached.

[0040] Advantageously, the determination of an unscaled depth image can be trained in such a way that it is generally applicable to acquired monocular image data.

[0041] The depth data acquisition device according to the invention comprises a monocular image acquisition unit for the automated acquisition of image data from an environment of a 202505294

[0042] 7

[0043] Rail vehicle by means of a monocular image recording unit carried by the rail vehicle.

[0044] The depth data determination device according to the invention has a depth estimation unit for determining an unscaled depth map by means of a KL-based depth estimation method.

[0045] Part of the depth data acquisition device according to the invention is also a model unit for generating a track model of the track currently being traveled by the rail vehicle with an absolute scale.

[0046] Furthermore, the depth data determination device according to the invention includes a projection unit for projecting the railway track model onto the unscaled depth map.

[0047] The depth data acquisition device according to the invention comprises an image data generation unit for generating a scaled depth map based on the unscaled depth map and on the absolute scale of the track model projected onto the unscaled depth map. The depth data acquisition device according to the invention shares the advantages of the inventive method for determining a scaled depth image with monocular image acquisition combined with track path data.

[0048] The object detection device according to the invention includes a depth data determination device according to the invention for determining a scaled depth map of the environment of a rail vehicle.

[0049] The object detection device according to the invention also comprises an object detection unit for identifying an object in the scaled depth map and for locating the object based on depth information encompassed by the scaled depth map, whereby object detection data is generated. The object detection device according to the invention shares the advantages of the object detection method according to the invention.

[0050] The rail vehicle according to the invention comprises a drive device for propelling a driving movement of the rail vehicle, a braking device for braking a driving movement of the rail vehicle, an object detection unit according to the invention-202505294

[0051] 8

[0052] direction for identifying and locating an object in the vicinity of the rail vehicle and a control device for controlling the drive device and the brake device depending on an identification and localization of an object by the object detection device.

[0053] A large part of the aforementioned components of the depth data acquisition device and the object detection device according to the invention can be implemented wholly or partially in the form of software modules in a processor of a corresponding computer system, e.g. in a control unit or a computer system of a rail vehicle.

[0054] A largely software-based implementation has the advantage that even previously used computing systems can be easily retrofitted by a software update to operate in the manner according to the invention.Therefore, the problem is also solved by a corresponding computer program product with a computer program that can be directly loaded into a computing system, with program sections to execute the steps of the inventive method for determining a scaled depth image with monocular image acquisition combined with rail path data, in particular the steps of determining an unscaled depth map by means of an AI-based depth estimation method, generating a track model of the track currently being traveled by the rail vehicle with an absolute scale, projecting the track model onto the unscaled depth map and generating a scaled depth map based on the unscaled depth map and on the basis of the absolute scale of the track model projected onto the unscaled depth map, when the program is executed in the computing system.In addition to the computer program itself, such a computer program product may include additional components such as documentation and / or additional components, as well as hardware components such as hardware keys (dongles, etc.) for using the software.

[0055] For transport to the computer system or control unit and / or for storage on or in the computer system or control unit, a computer-readable medium, e.g., a memory stick, a hard drive, or other portable or permanently installed data carrier, can be used, on which the program sections of the computer program that can be read and executed by a computer system are stored. (202505294)

[0056] 9

[0057] For example, a computer system can have one or more cooperating microprocessors or the like.

[0058] The dependent claims and the subsequent description each contain particularly advantageous embodiments and further developments of the invention. In particular, the claims of one claim category may also be further developed analogously to the dependent claims of another claim category and their descriptive parts. Furthermore, within the scope of the invention, the various features of different embodiments and claims may also be combined to form new embodiments.

[0059] In a variant of the inventive method for determining a scaled depth image using monocular image acquisition combined with rail path data, projecting the rail track model onto the unscaled depth map comprises segmenting a rail track in the acquired monocular image data or the unscaled depth map, whereby segmented image data with a segmented rail track are obtained, and adapting the rail track model to the segmented image data with the segmented rail track. By means of image processing or object recognition in image data, in particular computer vision and / or edge detection, the rails are identified and located in the acquired monocular image data or the unscaled depth data. The segmentation delineates a sub-area in the monocular image data or unscaled depth data that is occupied by the rails.The preferably parameterizable track model is then adapted to fit into this sub-area. Since an absolute scale is known for the track model, different distances to the train can now be assigned to different image areas.

[0060] The railway track model can be defined as a flexible track model consisting of segments that are flexibly adapted to the segmented track alignment when projected onto the unscaled depth map. Alternatively, the railway track model can also be a polynomial or clothoid, which can be parameterized and whose parameters can be adapted to the existing preprocessed image data or the preprocessed unscaled depth data. The preprocessed data should include segmented and / or edge-filtered data.

[0061] 10

[0062] If additional map information and information about the ego position of the rail vehicle and its orientation are available, a track model can also be generated based on the map information and adapted to the segmented image data or depth data.

[0063] Preferably, the assignment of the absolute scale of the track model to points in the unscaled depth data involves a non-linear regression of the scale based on support points of the adapted track model in the unscaled depth image, the unscaled depth image data, or the unscaled depth map. The support points are correspondences between the metric distances of the track model, the pixels in the monocular image data, and the corresponding pixels in the unscaled depth images, depth image data, or the unscaled depth map.

[0064] The reference values ​​are derived from the projection of the track model (consisting of its 3D points) onto the depth image or the unscaled depth map. This provides both the distance of the 3D point to the camera or the monocular image acquisition unit and the unscaled distance from the unscaled depth map. The absolute scale can then be calculated from these values. Advantageously, the absolute scale can be extended beyond the reference values ​​to cover the entire image area.

[0065] Preferably, the non-linear regression is based on a polynomial scaling function, the independent variables of which are determined. Advantageously, the polynomial scaling can also be used to model non-linear behavior of the depth distances shown in the depth image data. Advantageously, the regression for determining the scaling function is designed to be robust against outliers, so that a valid scaling function can be determined even if image occlusions by obstacles in the area of ​​the rails occur.

[0066] Preferably, the adaptation of the track model to the image data or unscaled depth data, or the unscaled depth map, is based on a model of the track traversed by the rail vehicle and the track bed encompassing the track. The alignment during the adaptation of the track model to the monocular image data or the unscaled depth data, or the unscaled depth map, is therefore preferably based on the rails captured in the acquired image data and on additional image information, in particular about the rail-202505294

[0067] 11

[0068] Preferably, in addition to the rails to be traversed, adjacent or more distant rails are also included. To project the track model onto the unscaled depth data or the unscaled depth map and to base the absolute scale on a particularly broad data set, adjacent or more distant rails that are captured by the monocular imaging unit can also be taken into account.

[0069] Preferably, the track model used to determine the absolute scale includes the rails of the track currently being traveled by the rail vehicle and their track bed. Advantageously, rails are used that typically traverse large portions of the unscaled depth map, making them suitable as an absolute scale.

[0070] Preferably, the independent parameters of the scaling function are continuously tracked during a single trip or over many trips. This tracking is advantageously used to support the robustness of the regression in the event of changes to the track layout or extrinsic or intrinsic camera parameters.

[0071] Additionally, ego-position data of the rail vehicle can be determined to define a track model based on highly accurate map data and the knowledge of the rail vehicle's absolute position within that data. The knowledge of the rail vehicle's absolute position, combined with map data, is advantageous for defining a track model because the curvature of the rails at a known location is determined based on the map data.

[0072] Knowledge of the ego position can also be advantageously used to determine an absolute position in the captured image data in order to locate individual objects and to identify these objects based on their absolute position by comparison with map data.

[0073] The ego position data of the rail vehicle is preferably determined based on satellite navigation data, preferably RTK-based satellite navigation data. Satellite navigation data, especially RTK-based satellite navigation data, can be determined with particular accuracy, resulting in highly precise positioning on the scaled depth map.

[0074] 12

[0075] The real-time kinematics (RTK) method, which is the preferred method for determining Ego position data, utilizes not only code phase measurement but also carrier phase measurement of the GNSS signals (GNSS stands for Global Navigation Satellite System). Signal processing involves not only evaluating the information content of the carrier signal but also determining its exact phase, i.e., the precise moment within the oscillation cycle. Since the wavelength of the carrier signal is significantly shorter than the wavelength of the data it contains, carrier phase measurement enables a much more precise distance determination. A reference station, whose position is precisely known, continuously measures the carrier phases of the received satellite signals and compares them with the expected values ​​based on its known position.The discrepancies, primarily caused by atmospheric influences and multipath effects, are then transmitted in real time as correction data to the RTK-enabled receiving unit, in this case, a receiver unit in the drone. A unit for determining the ego position of the rail vehicle, which is preferably an integral part of the rail vehicle, applies these corrections to its own measurements to determine its ego position and can thus determine its position with centimeter accuracy.

[0076] The data for the self-localization of the camera or image acquisition unit of the rail vehicle can be acquired by very different types of measuring devices. The measurement data mentioned can include the following types of measurement data:

[0077] - Satellite navigation data,

[0078] - IMU data (IMU = inertial measurement unit)

[0079] - Speed ​​sensor data,

[0080] - Odometry data,

[0081] - so-called RTK correction data.

[0082] An inertial measurement unit measures acceleration and deceleration data and can be used to determine the orientation of a rail vehicle and thus to determine the pose of the rail vehicle.

[0083] Speed ​​sensor data and odometry data can be used to determine the ego position of a rail vehicle. 202505294

[0084] 13

[0085] Satellite navigation can achieve high accuracy for position or direction determination within the framework of differential satellite navigation by supplementing it with reference stations and evaluating the phase shift of the carrier wave. An inertial measurement unit can be used for orientation measurement. Velocity and odometry data can also be used for position determination. Combining different measurement methods is also advantageous. Depending on the situation and the current applicability of the various methods, the measured values ​​from different measurement techniques can be combined, for example, through data fusion, particularly through weighted averaging of the measured values.

[0086] The unscaled depth map, as already mentioned, is preferably generated by an artificial neural network based on the principle of deep learning, which is applied to the acquired image data. For this purpose, the MDE method already mentioned in the introductory description and supported by literature is preferably used.

[0087] In one variant of the captured monocular image data, this monocular image data includes grayscale data. Grayscale data can be subjected to a mobile data entry (MDE) procedure or object recognition with relatively little data processing effort.

[0088] The invention is explained in more detail below with reference to the accompanying figures and exemplary embodiments. The figures show:

[0089] FIG 1 shows a schematic representation of a process for determining depth data in monocularly acquired image data of the environment of a rail vehicle within the framework of a method according to an embodiment of the invention.

[0090] FIG 2 shows a flowchart illustrating a method for determining a scaled depth image using monocular image acquisition combined with rail path data according to an embodiment of the invention.

[0091] FIG 3 shows a flowchart illustrating a method for training an AI-based depth estimation method according to an embodiment of the invention, 202505294

[0092] 14

[0093] FIG 4 shows a schematic representation of a depth data acquisition device according to an embodiment of the invention,

[0094] FIG 5 shows a schematic representation of an object detection device according to an embodiment of the invention,

[0095] FIG 6 shows a schematic representation of a rail vehicle according to an embodiment of the invention.

[0096] Figure 1 illustrates a schematic representation of scenario 1 of a process for determining depth data in monocularly acquired image data BD of the environment of a rail vehicle 2 within the framework of a method according to an embodiment of the invention. Scenario 1 shown in Figure 1 comprises a rail vehicle 2 with a monocular image acquisition unit 3. The rail vehicle 2 travels on a rail track 4 and acquires monocular image data BD of its environment, which in particular comprises the rail track 4.

[0097] The monocular image data BD are transmitted to an artificial neural network DNN based on deep learning. The artificial neural network DNN determines an unscaled depth map TKU using a AI-based depth estimation method. This procedure corresponds to step 2.II, which is illustrated in the flowchart shown in FIG. 2.

[0098] To obtain a scaled depth map (TKS), railway track data (RTI) is now acquired from an additional data source. This railway track data (RTI) comprises a railway track model (M), which is adapted to the existing monocular image data using a regression process, corresponding to step 2.13 in flowchart 200 shown in FIG. 2.

[0099] In this way, the railway line model M is projected, to put it simply, into the unscaled depth map TKU, thereby creating an absolute depth scale and thus generating a scaled depth map TKS.

[0100] The diagram shown in FIG 1 illustrates the aforementioned scenario 1. The monocular image data BD acquired by the monocular image acquisition unit 3, which are generated, for example, in color or as a grayscale image, are processed by a pre-202505294

[0101] 15

[0102] A trained artificial neural network (DNN), based on deep learning, processes the data into an unscaled depth map (TKU), which is then scaled in a further step using track information or rail route data (RTI). For this purpose, the track model (M) is used, and the position of the track (4) relative to the camera (3) can also be provided. The rail route model (M) can also include the track bed. Both the track and the track bed are projected onto the depth image or unscaled depth map (TKU). The corresponding unscaled depth values ​​in the depth image are determined based on the known track depth values. In a nonlinear regression, the independent variables of a (polynomial) scaling function are determined.The scaling function is then applied to the entire depth image generated by the artificial neural network (DNN), resulting in a metrically accurate depth image or scaled depth map (TKS). The DNN used for depth image generation was pre-trained using self-supervised deep learning. Alternatively, supervised training of the artificial neural network can be used. The scale, which is ultimately used for the accurate depth image or scaled depth map (TKS), is not considered during the training of the DNN.

[0103] Thus, the artificial neural network DNN can be used independently of the type of image acquisition unit 3 and its position within the rail vehicle 2. The track information or rail route data (RTI) can be provided in various ways. Location determination (see Determining the Ego Position of the Rail Vehicle) is possible, and the shape of the track is preferably captured directly via image processing (running in parallel on the system). An existing localization system that determines the rail vehicle's location using a map can also be used.

[0104] Adjacent rails can also be used to project the absolute scale onto the unscaled image data. Image acquisition unit 3 is calibrated both intrinsically and extrinsically. It should be noted that precise extrinsic calibration is neither necessary nor required for adapting the track model to the preprocessed image data or preprocessed unscaled depth image data. The exact extrinsic calibration is determined precisely through this adaptation. The regression for determining the scaling function is designed to be robust against outliers, ensuring that a valid scaling function is maintained even with partial obstructions (by obstacles).

[0105] 16

[0106] A scaling function may be present. Furthermore, this allows potential obstacles in the track bed to be detected during travel using a regression process.

[0107] FIG 2 shows a flowchart 200 illustrating a method for determining a scaled depth image using monocular image acquisition combined with rail path data according to an embodiment of the invention.

[0108] In the inventive method for determining a scaled depth image or a scaled depth map TKS with monocular image acquisition combined with rail path data, in step 2.1 image data BD from an environment U of a rail vehicle 2 are automatically acquired by a monocular image acquisition unit 3 carried by the rail vehicle 2.

[0109] In order to obtain unscaled depth data, an unscaled depth map TKU is determined in step 2.II using a computer-based depth estimation method TSV.

[0110] To generate an absolute scale, a track model M of the track 4 currently being used by the rail vehicle 2 is generated in step 2.III with an absolute scale AS.

[0111] Subsequently, in step 2.IV, the railway track model M is projected onto the unscaled depth map TKU.

[0112] In step 2.V, a scaled depth image or a scaled depth map TKS is generated based on the unscaled depth map TKU and on the basis of the absolute scale AS of the railway track model M projected into the unscaled depth map TKU with an absolute depth scale.

[0113] Figure 3 shows a flowchart 300 illustrating a method for training an AI-based depth estimation method TSV according to an embodiment of the invention. In the embodiment described in Figure 3, supervised training is performed.

[0114] In step 3.I, monocular training image data (TBD), labelled with a depth estimate (TS), is received. The TBD training image data comprises the track traversed by a rail vehicle and its surroundings. 202505294

[0115] 17

[0116] As a test, in step 3.II, depth information TI is then estimated as result data based on the monocular training image data TBD using the AI-based depth estimation method TSV itself.

[0117] In step 3.III, an adapted AI-based depth estimation method, ATSV, is generated based on the determined depth information TI and the depth estimate TS of the respective label in the training image data to provide feedback on the test performed in step 3.II. Steps 3.II and 3.III are ideally repeated with the respective adapted AI-based depth estimation method ATSV until a predetermined quality criterion for the depth information estimated by the adapted AI-based depth estimation method ATSV is met.

[0118] The AI-based depth estimation method ATSV, thus trained and adapted, can then be used in step 2.II of the procedure illustrated in FIG 2 for determining a scaled depth image with monocular image acquisition combined with rail path data, in which unscaled depth maps TKU are determined by an AI-based depth estimation method TSV.

[0119] FIG 4 shows a schematic representation of a depth data acquisition device 40 according to an embodiment of the invention.

[0120] The depth data acquisition device 40 comprises a monocular image acquisition unit 3 for the automated acquisition of image data BD from the environment of a rail vehicle 2 by a monocular image acquisition unit 3 carried by the rail vehicle 2.

[0121] The depth data acquisition device 40 includes a depth estimation unit 41 for determining an unscaled depth map TKU using an AI-based depth estimation method TSV. The AI-based depth estimation method is generated by a deep-learning-based artificial neural network, which is trained using the method illustrated in FIG. 3.

[0122] Part of the depth data acquisition device 40 is a model unit 42 for generating a track model M of the track currently being traveled by the rail vehicle 2 with an absolute scale AS.202505294

[0123] 18

[0124] The depth data determination device 40 according to the invention also has a projection unit 43 for projecting the railway track model M onto the unscaled depth map TKU.

[0125] The depth data determination device 40 according to the invention also comprises an image data generation unit 44 for generating a scaled depth map TKS based on the unscaled depth map TKU and on the basis of the absolute scale AS of the railway track model M projected into the unscaled depth map TKU.

[0126] Figure 5 shows a schematic representation of an object detection device 50 according to an embodiment of the invention. The object detection device 50 shown in Figure 5 comprises the depth data acquisition device 40 illustrated in Figure 4 and an object detection unit 51, which detects an object in the scaled depth map TKS by applying image processing, preferably computer vision, and locates it based on the depth information TI present in the scaled depth map TKS and outputs object detection data OD.

[0127] Figure 6 shows a schematic representation 60 of a rail vehicle 2 according to an embodiment of the invention. The rail vehicle 2 illustrated in Figure 6 includes the object detection device 50 shown in Figure 5. The object detection device 50 can detect and locate objects in the area of ​​a railway track 4 in use.

[0128] Finally, it should be noted once again that the methods and devices described above are merely preferred embodiments of the invention and that the invention can be varied by a person skilled in the art without departing from the scope of the invention, insofar as it is defined by the claims. For the sake of completeness, it should also be noted that the use of the indefinite articles "a" or "an" does not preclude the possibility that the features in question may be present multiple times. Likewise, the term "unit" does not preclude the possibility that it consists of several components, which may also be spatially distributed.

[0129] Regardless of the grammatical gender of a given term, persons of male, female, or other gender identities are included. 202505294

[0130] 19

[0131] Reference symbol list

[0132] 1 Scenario of a process for determining depth data

[0133] 2 Rail vehicle

[0134] 3 monocular image acquisition units

[0135] 4 Railway line

[0136] 40 Depth data acquisition device

[0137] 41 Depth Estimation Unit

[0138] 42 model unit

[0139] 43 projection units

[0140] 44 Image data generation unit

[0141] 50 object detection devices

[0142] 51 object detection unit

[0143] 60 Schematic representation of a rail vehicle

[0144] AS absolute scale

[0145] ATSV adapted AI-based depth estimation method

[0146] BD monocular image data

[0147] DNN (artificial neural network based on deep learning) M railway line model

[0148] OD object detection data

[0149] RTI rail route data

[0150] TBD monocular training image data

[0151] TI In-depth Information

[0152] TKS scaled depth map

[0153] TKU unscaled depth map

[0154] TS Depth Estimation

[0155] TSV Kl-based depth estimation method

[0156] U surroundings

Claims

202505294 20 Patent claims 1. Method for determining a scaled depth image using monocular image acquisition combined with rail path data, comprising the steps: - automated acquisition of monocular image data (BD) from an environment (U) of a rail vehicle (2) by a monocular image acquisition unit (3) carried by the rail vehicle (2), - Determining an unscaled depth map (TKU) using a computer-based depth estimation (TSV) method based on monocular image data (BD), - Generating a track model (M) of the track (4) currently being traveled by the rail vehicle (2) with an absolute scale (AS), - Projecting the railway model (M) onto the unscaled depth map (TKU), - Generating a scaled depth map (TKS) based on the unscaled depth map (TKU) and on the basis of the absolute scale (AS) of the railway model (M) projected onto the unscaled depth map (TKU).

2. The method of claim 1, wherein projecting the track model into the unscaled depth map (SDM) comprises segmenting a track in the unscaled depth map (SDM) or the monocular image data (MD) and adapting the track model (M) to the segmented track.

3. Method according to claim 1 or 2, wherein the generation of the scaled depth map (TKS) based on the railway model (M) comprises a non-linear regression of the absolute scale (AS) based on support values ​​of the adapted railway model (M) in the unscaled depth map (TKU).

4. Method according to claim 3, wherein the non-linear regression is performed on the basis of a polynomial scaling function, the independent variables of which are determined.

5. Method according to one of the preceding claims, wherein the track model (M) comprises the rails of the track (4) currently being traveled on by the rail vehicle (2) and the track bed.

6. A method according to any of the preceding claims, wherein the track model (M) includes, in addition to the rails traveled by the rail vehicle (2), also the rails adjacent to the rails traveled. 21 7. Method according to one of the preceding claims, wherein Ego position data of the rail vehicle (2) are determined and used in the steps for generating a track model (M) and / or for projecting the track model (M) onto the unscaled depth map (TKU).

8. Method according to claim 7, wherein the ego position data of the rail vehicle (2) are determined on the basis of satellite navigation data, preferably RTK-based satellite navigation data.

9. Method for training an AI-based depth estimation method, which is to be applied within the framework of the method according to any one of claims 1 to 8, comprising the steps: - Receiving monocular training image data (TBD) labelled with a depth estimate (TS), which includes rails traveled by a rail vehicle and their surroundings, - Determining depth information (TI) based on monocular training image data (TBD) using the AI-based depth estimation method (TSV), - Generating an adapted AI-based depth estimation method (ATSV) based on the determined depth information (TI) and on the basis of the depth estimation (TS) of the respective label of the training image data (TBD).

10. Method for object detection, comprising the steps: - Performing the method according to any one of claims 1 to 8, wherein a scaled depth map (DTM) of the environment of a rail vehicle (2) is determined, - Identifying an object in the scaled depth map (DTM), - Locating the object based on depth information included in the scaled depth map (TKS).

11. Depth data acquisition device (40), comprising: - a monocular image acquisition unit (3) for the automated acquisition of monocular image data (BD) from an environment (U) of a rail vehicle (2) by a monocular image acquisition unit (3) carried by the rail vehicle (2), - a depth estimation unit (41) for determining an unscaled depth map (DCM) by a KL-based depth estimation method (TSV),202505294 22 - a model unit (42) for generating a track model (M) of the track (4) currently being traveled by the rail vehicle (2) with an absolute scale (AS), - a projection unit (43) for projecting the railway line model (M) onto the unscaled depth map (TKU), - an image data generation unit (44) for generating a scaled depth map (TKS) based on the unscaled depth map (TKU) and on the basis of the absolute scale (AS) of the railway line model (M) projected onto the unscaled depth map (TKU).

12. Object detection device (50), comprising: - a depth data acquisition device (40) according to claim 11, for determining a scaled depth map (DCM) of the environment of a rail vehicle (2), - an object detection unit (51) for identifying an object in the scaled depth map (TKS) and for locating the object based on depth information encompassed by the scaled depth map (TKS), generating object detection data (OD).

13. Railway vehicle (2), comprising: - a drive device for propelling a movement of the rail vehicle (2), - a braking device for slowing down the movement of the rail vehicle, - an object detection device (50) according to claim 12 for identifying and locating an object in the vicinity of the rail vehicle, - a control device for controlling the drive device and the brake device depending on the identification and localization of an object by the object detection device (50).

14. Computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of determining an unscaled depth map (SDM) by means of an AI-based depth estimation method (DSM), generating a track model (M) of the track (4) currently traversed by the rail vehicle (2) with an absolute scale (AS), projecting the track model (M) onto the unscaled depth map (SDM), and generating a scaled depth map (SDM) based on the unscaled depth map (SDM) and on the basis of the absolute scale (AS) of the track model (M) projected onto the unscaled depth map (SDM) according to any one of claims 1 to 8. 23 15. Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of determining an unscaled depth map (SDM) by means of an AI-based depth estimation method (DSM), generating a track model (M) of the track (4) currently being traveled by the rail vehicle (2) with an absolute scale (AS), projecting the track model (M) onto the unscaled depth map (SDM) and generating a scaled depth map (SDM) based on the unscaled depth map (SDM) and on the basis of the absolute scale (AS) of the track model (M) projected onto the unscaled depth map (SDM) according to any one of claims 1 to 8.