Railway vehicle control apparatus and method
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- POSCO HLDG INC
- Filing Date
- 2024-12-18
- Publication Date
- 2026-06-18
Smart Images
Figure KR2024097123_18062026_PF_FP_ABST
Abstract
Description
Railway vehicle control device and method
[0001] The present embodiments relate to a railway vehicle control device and method.
[0002] Various autonomous braking and collision avoidance technologies have been developed to ensure the safe operation of railway vehicles in railway systems. Existing technologies primarily utilize a method in which sensors installed at the front detect an object when it enters a danger zone and apply immediate braking based on this detection.
[0003] However, because existing technology reacts only after an object has already entered the danger zone, situations where collisions cannot be avoided even with real-time braking can frequently occur due to the long braking distances of railway vehicles. This can be seen as a technical limitation that does not account for long braking distances.
[0004] Therefore, technology to warn railway vehicles in advance before entering danger zones is necessary, but development in such technology is insufficient.
[0005] These embodiments can provide a railway vehicle control device and method capable of tracking the movement of an object around a danger zone and predicting the trajectory of the object to recognize and prevent the risk of collision in advance.
[0006] In one aspect, the present embodiments may provide a railway vehicle control device comprising: a receiving unit that receives image data, LiDAR data, and driving information of a railway vehicle using one or more sensors of a railway vehicle; an object trajectory calculation unit that recognizes an object and a track area based on the image data and LiDAR data and generates an object trajectory which is the path the object has traveled; a driving trajectory prediction unit that generates a driving trajectory which is the driving trajectory expected to be traveled by the railway vehicle for a specific period of time based on the track area and the driving information of the railway vehicle; an object trajectory prediction unit that generates an object trajectory which is the path the object is expected to move for a specific period of time based on the object trajectory and the driving information of the railway vehicle; a collision risk determination unit that determines the risk of collision between the railway vehicle and the object by comparing the driving trajectory and the object trajectory; and a control unit that controls the railway vehicle to perform a warning operation according to the determination result of the collision risk determination unit.
[0007] In another aspect, the present embodiments may provide a railway vehicle control method comprising: a receiving step of receiving image data, LiDAR data, and driving information of a railway vehicle using one or more sensors of a railway vehicle; an object trajectory calculation step of recognizing an object and a track area based on the image data and LiDAR data and generating an object trajectory which is the path the object has traveled; a driving trajectory prediction step of generating a driving trajectory which is the driving trajectory expected to be traveled by the railway vehicle for a specific period of time based on the track area and the driving information of the railway vehicle; an object trajectory prediction step of generating an object trajectory which is the path the object is expected to move for a specific period of time based on the object trajectory and the driving information of the railway vehicle; a collision risk determination step of determining a collision risk between the railway vehicle and the object by comparing the driving trajectory and the object trajectory; and a control step of controlling the railway vehicle to perform a warning operation according to the result of the determination step of the collision risk determination step.
[0008] According to the embodiments, a railway vehicle control device and method can be provided that can track the movement of an object around a danger zone and predict the trajectory of the object to recognize and prevent the risk of collision in advance.
[0009] FIG. 1 is a block diagram illustrating a railway vehicle according to the embodiments.
[0010] FIG. 2 is a block diagram illustrating a railway vehicle control device according to the embodiments.
[0011] FIG. 3 is a drawing for explaining a receiver according to the embodiments.
[0012] FIG. 4 is a diagram illustrating the generation of object feature data according to the embodiments.
[0013] FIG. 5 is a diagram illustrating the operation of a collision risk determination unit according to the embodiments.
[0014] FIG. 6 is a flowchart for explaining a railway vehicle control method according to the embodiments.
[0015] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the exemplary drawings. In assigning reference numerals to the components of each drawing, the same components may have the same reference numeral as much as possible, even if they are shown in different drawings. Furthermore, in describing the embodiments, if it is determined that a detailed description of related known components or functions may obscure the essence of the technical concept, such detailed description may be omitted. Where terms such as "comprising," "having," or "consisting of" are used in this specification, other parts may be added unless "only" is used. Where a component is expressed in the singular, it may include a plural unless otherwise specified.
[0016] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used to describe the components of the present disclosure. These terms are used merely to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by such terms.
[0017] In describing the positional relationship of components, where it is stated that two or more components are "connected," "combined," or "joined," it should be understood that while the two or more components may be directly "connected," "combined," or "joined," they may also be "connected," "combined," or "joined" with other components "intervened." Here, the other components may be included in one or more of the two or more components that are "connected," "combined," or "joined" with one another.
[0018] In describing the temporal flow relationship regarding components, methods of operation, or methods of production, for example, when the temporal or sequential relationship is described using "after," "following," "next," or "before," it may include cases where the relationship is not continuous unless "immediately" or "directly" is used.
[0019] Meanwhile, where numerical values or corresponding information regarding a component (e.g., levels, etc.) are mentioned, even without separate explicit notation, the numerical values or corresponding information may be interpreted as including a range of error that may occur due to various factors (e.g., process factors, internal or external shocks, noise, etc.).
[0020] In the present disclosure, a railway vehicle may refer to a vehicle manufactured for the purpose of operating on a railway track. That is, a railway vehicle may be substituted with other terms having equivalent meanings, such as train, locomotive, power car, passenger car, freight car, special car, etc., and is not limited to a specific vehicle as long as it is a vehicle operating on a railway track. In particular, the railway vehicle described in the present disclosure is described on the premise of an autonomous locomotive, but is not limited thereto, and may be applied substantially the same to non-autonomous vehicles as long as it does not contradict the technical concept of the present disclosure. Furthermore, in the present disclosure, a railway vehicle may include a locomotive equipped with a torpedo ladle car (TLC), but is not limited thereto.
[0021] In the present disclosure, the term "railway track" may be replaced with other terms having equivalent meanings, such as track, railway, running track, etc. Additionally, the term "image" may be replaced with other terms having equivalent meanings, such as picture, frame, image, image data, image data, etc.
[0022] FIG. 1 is a block diagram illustrating a railway vehicle according to the embodiments.
[0023] Referring to FIG. 1, a railway vehicle (1) according to the embodiments may include a railway vehicle control device (100) and a driving device (200). The railway vehicle control device (100) and the driving device (200) may be connected to each other via communication.
[0024] A railway vehicle control device (100) can recognize objects detected on the track where the railway vehicle travels and around the vehicle. To this end, the railway vehicle control device (100) can receive image data through any one of an RGB camera, a thermal camera, or an infrared camera installed on the front, rear, right, and left sides of the railway vehicle. Additionally, the railway vehicle control device (100) can receive LiDAR data through LiDAR sensors installed on the front, rear, right, and left sides of the railway vehicle. In this case, the railway vehicle control device (100) can detect all track areas and objects within the image using the image data and LiDAR data.
[0025] Subsequently, the railway vehicle control device (100) can recognize an object and a track area based on image data and LiDAR data, generate a trajectory of the object, and generate a driving prediction trajectory based on the track area and the driving information of the railway vehicle. The railway vehicle control device (100) can generate a driving prediction trajectory based on the trajectory of the object and the driving information of the railway vehicle, compare the driving prediction trajectory and the object prediction trajectory to determine the risk of collision between the railway vehicle and the object, and perform control of the driving of the railway vehicle so that the railway vehicle performs an action according to the determination result.
[0026] This railway vehicle control device (100) will be described in more detail below with reference to the relevant drawings.
[0027] In one example, the railway vehicle control device (100) may directly transmit a control command to the running device (200) of the railway vehicle. Alternatively, in another example, the railway vehicle control device (100) may transmit a judgment result or a control command to a separate control device (not shown) that controls the running device (200) of the railway vehicle, to display a warning message to the railway vehicle, control the deceleration of the railway vehicle, or apply emergency braking to the railway vehicle.
[0028] According to one example, when the railway vehicle is in autonomous driving mode, the railway vehicle control device (100) can automatically transmit a control command for the driving of the railway vehicle. If the railway vehicle is in manual driving mode, the railway vehicle control device (100) can output a control alarm for the driving of the railway vehicle through a separate output device. In this case, the driver can control the driving of the railway vehicle by looking at the alarm.
[0029] The driving device (200) is a device that directly drives a railway vehicle and may include at least one of a wheel, a suspension device, a braking device, a power device, and an auxiliary device. The driving device (200) is not limited to a specific device, provided that it can display a warning message to the railway vehicle, control the deceleration of the railway vehicle, or perform emergency braking of the railway vehicle in accordance with a control command of the railway vehicle control device (100).
[0030] Hereinafter, a railway track monitoring device and method according to embodiments of the present disclosure will be described with reference to the attached drawings.
[0031] FIG. 2 is a block diagram illustrating a railway vehicle control device according to the present embodiments. FIG. 3 is a diagram illustrating a receiver according to the present embodiments. FIG. 4 is a diagram illustrating the generation of object feature data according to the present embodiments. FIG. 5 is a diagram illustrating the operation of a collision risk determination unit according to the present embodiments.
[0032] Referring to FIG. 2, the railway vehicle control device (100) may include a receiving unit (110) that receives image data, LiDAR data, and driving information of the railway vehicle using one or more sensors of the railway vehicle; an object trajectory calculation unit (120) that recognizes an object and a track area based on the image data and LiDAR data and generates an object trajectory, which is the path the object has traveled; a driving trajectory prediction unit (130) that generates a driving trajectory, which is the driving trajectory expected to be traveled by the railway vehicle for a specific time based on the track area and the driving information of the railway vehicle; an object trajectory prediction unit (140) that generates an object trajectory, which is the path the object is expected to move for a specific time based on the object trajectory and the driving information of the railway vehicle; a collision risk judgment unit (150) that determines the risk of collision between the railway vehicle and the object by comparing the driving trajectory and the object trajectory; and a control unit (160) that controls the railway vehicle to perform a warning operation according to the judgment result of the collision risk judgment unit.
[0033] Referring to FIG. 2, the receiver (110) can receive image data, LiDAR data, and driving information of the railway vehicle using one or more sensors of the railway vehicle.
[0034] For example, one or more sensors of a railway vehicle may include one or more of a front camera (111), a rear camera (112), a right camera (114), and a left camera (113) capable of receiving image data.
[0035] The front camera (111), rear camera (112), right camera (114), and left camera (113) are implemented using any one of an RGB camera, a thermal imaging camera, an infrared camera, a CCD (Charge-Coupled Device) image sensor, or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, so that the external environment of the railway vehicle can be captured to obtain images of the track and surroundings in real time. However, this is merely an example, and is not limited to a specific device as long as images of the track and surroundings are obtained in real time and the technical concept of the present disclosure can be applied.
[0036] In this case, the image data may include the shape or contour of the track area and object contained in the surrounding image. Additionally, the image data may include the color and texture of the track area and object contained in the surrounding image. Furthermore, if the image data corresponds to a continuous series of images over time, the speed and direction of the object can be estimated using the image data.
[0037] As another example, it may include one or more of a front lidar (115), a rear lidar (117), a right lidar (116), and a left lidar (not shown) capable of receiving lidar data.
[0038] It may include one or more of a front lidar (115), a rear lidar (117), a right lidar (116), and a left lidar (not shown) capable of receiving lidar data.
[0039] The front lidar (115), rear lidar (117), right lidar (116) and left lidar (not shown) can receive lidar data in the form of a three-dimensional point cloud (hereinafter, point cloud) and can obtain information about track areas and objects in the external environment of the railway vehicle in real time.
[0040] Each point included in the LiDAR data in the point cloud format can contain a location value. That is, LiDAR data can be expressed in the form of a point cloud represented in the form of three-dimensional coordinates (x,y,z).
[0041] However, this is merely an example, and is not limited to a specific device as long as LiDAR data regarding the surroundings of a driving track is acquired in real time and the technical concept of the present disclosure can be applied.
[0042] As another example, the receiver (110) can receive driving information of the railway vehicle.
[0043] For example, the driving information of a railway vehicle may include railway vehicle location information, which is the location of the railway vehicle received in real time via a GPS installed on the railway vehicle; railway vehicle speed information, which is the speed of the railway vehicle generated and received from the railway vehicle's speed sensor; and railway vehicle route information, which is information transmitted from a control tower and corresponds to the expected driving path of the railway vehicle.
[0044] As another example, the receiver (110) can acquire image data and lidar data continuously in real time, or acquire image data and lidar data according to a predetermined period or time.
[0045] In addition, the receiving unit (110) can store the driving information of the railway vehicle received in real time by mapping it to the image data and the lidar data, respectively.
[0046] Additionally, the receiving unit (110) can transmit image data around the driving track, lidar data, and driving information of the railway vehicle acquired in real time to the object trajectory calculation unit (120), driving trajectory prediction unit (130), and object trajectory prediction unit (140).
[0047] However, although the receiving unit (110) has been described as being included in the railway vehicle control device (100), it is not limited thereto and can be implemented as a separate device from the railway vehicle control device (100).
[0048] The object trajectory calculation unit (120) can recognize the object and track area based on image data and LiDAR data, and generate the trajectory of the object, which is the path the object traveled.
[0049] For example, objects may include obstacles, people, and railway facilities.
[0050] For example, the object trajectory calculation unit (120) can recognize an object and a track area by extracting image object feature data from image data, extracting lidar object feature data from lidar data, and using combined data generated by combining image object feature data and lidar object feature data.
[0051] The reason the object trajectory calculation unit (120) generates combined data is that the coordinate systems of the image data and the lidar data are different, and the types of data that can be received are different. Therefore, the object trajectory calculation unit (120) needs to convert the image data into image object feature data and unify the coordinate systems using the image object feature data and the lidar object feature data, or map the image object feature data and the lidar object feature data to each other to generate combined data. The operation of the object trajectory calculation unit (120) will be explained below.
[0052] Referring to FIG. 4, the object trajectory calculation unit (120) can extract image object feature data (122) from image data (121) using real-time object detection algorithms such as HOG (Histogram of Oriented Gradient), SIFT (Scale-Invariant Feature Transform), and deep learning models (Yolo, Faster R-CNN).
[0053] Additionally, the object trajectory calculation unit (120) can input the lidar data (123) into the RANSAC (Random Sample Consensus) technique or the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to group the point clouds and extract lidar object feature data (124) that distinguishes the objects.
[0054] Additionally, the object trajectory calculation unit (120) can classify objects through image object feature data (122) and combine them using weighted average, max / min fusion, confidence-based fusion techniques, and deep learning models such as FusionNet, PointNet, and DeepFusion, which can fuse three-dimensional position values of objects and track areas through LiDAR object feature data (124), and generate combined data (125).
[0055] In this case, the combined data (125) may include a recognized track area and track location information which is a three-dimensional location value of the track area. Additionally, the combined data (125) may include a recognized object and object location information which is a three-dimensional location value of the object.
[0056] Additionally, the object trajectory calculation unit (120) can calculate object velocity information of an object using a Kalman filter capable of tracking a state using time series data, an extended Kalman filter, an LSTM (Long Short-Term Memory) algorithm capable of predicting future data by learning based on past data, and a PointNet capable of classifying data from point cloud data. In this case, the combined data (125) may further include object velocity information.
[0057] However, not limited to the present embodiment, objects and track regions can be recognized through various methods, algorithms, and models capable of recognizing objects and track regions.
[0058] As another example, the object trajectory calculation unit (120) can generate the trajectory of an object using object location information and object velocity information included in the combined data (125).
[0059] For example, the object trajectory generation unit (120) can generate the trajectory of an object in real time by inputting object location information and object velocity information included in the combined data (125) into a Kalman filter, an extended Kalman filter, a deep learning algorithm such as LSTM, a Gated Recurrent Unit (GRU) optimized for sequential data processing, and a clustering algorithm (DBSCAN) used to identify groups by analyzing the density of the data.
[0060] However, the trajectory of an object can be generated through various methods, algorithms, and models capable of generating the trajectory of an object, not limited to the present embodiment.
[0061] Meanwhile, the driving trajectory prediction unit (130) can generate a driving prediction trajectory, which is a driving trajectory expected to be driven by a railway vehicle for a specific period of time, based on the track area and driving information of the railway vehicle.
[0062] For example, the driving trajectory prediction unit (130) can generate a predicted driving trajectory by inputting the track location information included in the track area, the railway vehicle location information, the railway vehicle speed information, and the railway vehicle path information included in the driving information of the railway vehicle into a preset driving trajectory algorithm.
[0063] For example, a specific time may be set to N seconds from the present (where N is a real number greater than 0). For example, the driving trajectory prediction unit may generate a driving trajectory that is expected to be driven by a railway vehicle for N seconds from the present. As another example, the specific time may be set to the time when the railway vehicle is expected to drive on a specific track based on track location information. However, the specific time may be set in various ways, not limited to the present embodiment.
[0064] As another example, as previously mentioned, the track location information may correspond to the three-dimensional location of the track existing in the direction in which the railway vehicle is traveling. In this embodiment, it may correspond to the current location of the track existing in the direction in which the railway vehicle is currently traveling. Additionally, as previously mentioned, the railway vehicle location information corresponds to the location of the railway vehicle, and in this embodiment, it may correspond to the current location of the railway vehicle currently traveling. Additionally, as previously mentioned, the railway vehicle speed information corresponds to the speed of the railway vehicle, and in this embodiment, it may correspond to the current driving speed of the railway vehicle currently traveling. Additionally, as previously mentioned, the railway vehicle path information may correspond to the expected driving path of the railway vehicle.
[0065] In this case, the pre-set driving trajectory algorithm may be an LSTM algorithm or a GAN (Generative Adversarial Network) algorithm.
[0066] However, the present embodiment is not limited to this, and the predicted driving trajectory can be generated through various methods, algorithms, and models capable of generating the predicted driving trajectory.
[0067] Meanwhile, the object trajectory prediction unit (140) can generate an object predicted trajectory, which is a path expected to be traveled by the object for a specific period of time, based on the trajectory of the object and the driving information of the railway vehicle.
[0068] For example, the object trajectory prediction unit (140) can generate an object predicted trajectory by inputting the object's trajectory and the railway vehicle's driving information into a preset object trajectory algorithm.
[0069] For example, the predicted object trajectory for a specific period of time can be generated in the same way as the driving trajectory prediction unit (130).
[0070] The embodiments of the present disclosure relate to a technology for preventing a railway vehicle from colliding with an object in advance. Accordingly, the collision risk determination unit described later determines whether there is a risk of collision by comparing the predicted object trajectory and the predicted driving trajectory. The determination of whether there is a risk of collision needs to be based on the same time period. Accordingly, the object trajectory prediction unit also needs to generate the predicted object trajectory for a specific time period in the same way as the driving trajectory prediction unit (130).
[0071] As another example, the railway vehicle speed information included in the railway vehicle driving information and the trajectory of the object generated by the object trajectory calculation unit (120) can be input into an algorithm that is pre-set as input values.
[0072] In this case, the pre-set object trajectory algorithm may include deep learning algorithms such as LSTM and GRU, and reinforcement learning algorithms such as DDPG (Deep Deterministic Policy Gradient) which can predict continuous state values.
[0073] Additionally, the preset algorithm may be an algorithm combined with either a Kalman filter or an extended Kalman filter used in the object trajectory calculation unit (120) and either an LSTM (Long Short-Term Memory) or a GRU (Gated Recurrent Unit).
[0074] However, not limited to the present embodiment, the object trajectory prediction unit (140) may utilize various methods, algorithms, and models as long as it can generate an object predicted trajectory.
[0075] Meanwhile, the collision risk determination unit (150) can determine the risk of collision between the railway vehicle and the object by comparing the expected driving trajectory and the expected object trajectory.
[0076] For example, the collision risk judgment unit (150) can determine that there is a risk of collision with an object if an overlapping section occurs between the expected driving trajectory and the expected object trajectory.
[0077] Referring to FIG. 5, there exists a point (151) where the object's expected trajectory (142) and the railway vehicle's expected trajectory (132) overlap. In this case, the collision risk judgment unit (150) can determine that there is a risk of collision with the object.
[0078] Alternatively, although the object (143)'s expected trajectory is not shown, if the object (143) remains stationary for a specific period of time and the expected driving trajectory (132) corresponds to a trajectory passing through the object (143), the collision risk determination unit (150) may determine that there is a risk of collision with the object.
[0079] However, the risk of collision with an object can be determined in various ways, not limited to the present embodiment.
[0080] Meanwhile, the control unit (160) can control the railway vehicle to perform a warning operation according to the judgment result of the collision risk judgment unit (150).
[0081] For example, a warning action of a railway vehicle may include one or more of the following: an action of displaying a warning message to the railway vehicle, an action of decelerating the railway vehicle, and an action of emergency braking the railway vehicle.
[0082] For example, a warning message on a railway vehicle may be displayed through an output device of the railway vehicle. In this case, the output device may include a display, an LED, and a speaker, and the output device may be installed inside or outside the railway vehicle.
[0083] As another example, if it is determined that there is a risk of collision with an object when the deceleration operation of a railway vehicle is controlled, the control unit (160) included in the railway vehicle control device (100) can transmit a control command to the vehicle's driving device (200) to control the deceleration operation of the railway vehicle.
[0084] As another example, if it is determined that there is a risk of collision with an object, the control unit (160) included in the railway vehicle control device (100) can transmit a control command to the vehicle's driving device (200) to control the railway vehicle to perform an emergency braking operation.
[0085] FIG. 6 is a flowchart for explaining a railway vehicle control method according to the embodiments.
[0086] A railway vehicle control method may include a receiving step of receiving image data, LiDAR data, and driving information of a railway vehicle using one or more sensors of a railway vehicle; an object trajectory calculation step of recognizing an object and a track area based on image data and LiDAR data and generating an object trajectory which is the path the object has traveled; a driving trajectory prediction step of generating a driving trajectory which is the driving trajectory expected to be traveled by the railway vehicle for a specific period of time based on the track area and driving information of the railway vehicle; an object trajectory prediction step of generating an object trajectory which is the path the object is expected to travel for a specific period of time based on the object trajectory and driving information of the railway vehicle; a collision risk judgment step of comparing the driving trajectory and the object trajectory to determine the collision risk between the railway vehicle and the object; and a control step of controlling the railway vehicle to perform a warning action according to the judgment result of the collision risk judgment step.
[0087] The receiving step can receive image data, LiDAR data, and driving information of the railway vehicle using one or more sensors of the railway vehicle. (S610)
[0088] For example, one or more sensors of a railway vehicle may include one or more of a front camera, a rear camera, a right camera, and a left camera capable of receiving image data.
[0089] The front camera, rear camera, right camera, and left camera are implemented using any one of an RGB camera, a thermal imaging camera, an infrared camera, a CCD (Charge-Coupled Device) image sensor, or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and can capture the external environment of the railway vehicle to acquire images of the running track and surroundings in real time. However, this is merely an example, and is not limited to a specific device as long as images of the running track and surroundings are acquired in real time and the technical concept of the present disclosure can be applied.
[0090] In this case, the image data may include the shape or contour of the track area and object contained in the surrounding image. Additionally, the image data may include the color and texture of the track area and object contained in the surrounding image. Furthermore, if the image data corresponds to a continuous series of images over time, the speed and direction of the object can be estimated using the image data.
[0091] As another example, it may include one or more of a front lidar, rear lidar, right lidar, and left lidar capable of receiving lidar data.
[0092] It may include one or more of a front lidar, rear lidar, right lidar, and left lidar capable of receiving lidar data.
[0093] The front lidar, rear lidar, right lidar, and left lidar can receive lidar data in the form of a three-dimensional point cloud (hereinafter, point cloud), and can acquire information about track areas and objects in the external environment of the railway vehicle in real time.
[0094] Each point included in the LiDAR data in the point cloud format can contain a location value. That is, LiDAR data can be expressed in the form of a point cloud represented in the form of three-dimensional coordinates (x,y,z).
[0095] However, this is merely an example, and is not limited to a specific device as long as LiDAR data regarding the surroundings of a driving track is acquired in real time and the technical concept of the present disclosure can be applied.
[0096] As another example, the receiving stage can receive driving information of a railway vehicle.
[0097] For example, the driving information of a railway vehicle may include railway vehicle location information, which is the location of the railway vehicle received in real time via a GPS installed on the railway vehicle; railway vehicle speed information, which is the speed of the railway vehicle generated and received from the railway vehicle's speed sensor; and railway vehicle route information, which is information transmitted from a control tower and corresponds to the expected driving path of the railway vehicle.
[0098] As another example, the receiving stage can acquire image data and lidar data continuously in real time, or acquire image data and lidar data according to a predetermined period or time.
[0099] In addition, the receiving stage can store real-time driving information of the railway vehicle by mapping it to image data and LiDAR data, respectively.
[0100] In addition, the receiving stage can transmit image data, LiDAR data, and driving information of the railway vehicle acquired in real time around the driving track to the object trajectory calculation stage, driving trajectory prediction stage, and object trajectory prediction stage.
[0101] However, although the receiving step has been described as being included in the railway vehicle control method, it is not limited thereto and can be implemented as a device separate from the railway vehicle control method.
[0102] The object trajectory calculation step recognizes object and track areas based on image data and LiDAR data, and can generate the object's trajectory, which is the path the object traveled. (S620)
[0103] For example, the object trajectory calculation step can recognize objects and track areas by extracting image object feature data from image data, extracting lidar object feature data from lidar data, and using combined data generated by combining image object feature data and lidar object feature data.
[0104] The reason the object trajectory calculation step generates combined data is that the coordinate systems of image data and LiDAR data are different, and the types of data that can be received are also different. Therefore, the object trajectory calculation step needs to convert image data into image object feature data and either unify the coordinate systems using the image object feature data and LiDAR object feature data, or map the image object feature data and LiDAR object feature data to each other to generate combined data. The operation of the object trajectory calculation step is explained below.
[0105] The object trajectory calculation step can extract image object feature data from image data using real-time object detection algorithms such as HOG (Histogram of Oriented Gradient), SIFT (Scale-Invariant Feature Transform), and deep learning models (Yolo, Faster R-CNN).
[0106] In addition, the object trajectory calculation step can input LiDAR data into the RANSAC (Random Sample Consensus) technique or the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to group point clouds and extract LiDAR object feature data that distinguishes objects.
[0107] In addition, the object trajectory calculation step can classify objects through image object feature data and combine them using weighted average, max / min fusion, confidence-based fusion techniques, and deep learning models such as FusionNet, PointNet, and DeepFusion, which can fuse 3D position values of objects and track areas through LiDAR object feature data, and generate combined data.
[0108] In this case, the combined data includes a recognized track area, wherein the track area may include track location information which is a 3D location value of the track area. Additionally, the combined data may include a recognized object and may include object location information which is a 3D location value of the object.
[0109] In addition, the object trajectory calculation step can calculate object velocity information using a Kalman filter, an extended Kalman filter, which can track the state using time-series data; an LSTM (Long Short-Term Memory) algorithm, which can predict future data by learning based on past data; and PointNet, which can classify data from point cloud data. In this case, the combined data may further include object velocity information.
[0110] However, not limited to the present embodiment, objects and track regions can be recognized through various methods, algorithms, and models capable of recognizing objects and track regions.
[0111] As another example, the object trajectory calculation step can generate the trajectory of an object using object location information and object velocity information included in the combined data.
[0112] For example, the object trajectory generation step can generate the object trajectory in real time by inputting the object location information and object velocity information included in the combined data into a Kalman filter, an extended Kalman filter, a deep learning algorithm called LSTM, a Gated Recurrent Unit (GRU) optimized for sequential data processing, and a clustering algorithm (DBSCAN) used to identify groups by analyzing the density of the data.
[0113] However, the trajectory of an object can be generated through various methods, algorithms, and models capable of generating the trajectory of an object, not limited to the present embodiment.
[0114] Meanwhile, the driving trajectory prediction step can generate a driving prediction trajectory, which is the driving trajectory expected to be traveled by the railway vehicle for a specific period of time, based on the track area and the driving information of the railway vehicle. (S630)
[0115] For example, the driving trajectory prediction step can generate a predicted driving trajectory by inputting track location information included in the track area, railway vehicle location information, railway vehicle speed information, and railway vehicle path information included in the driving information of the railway vehicle into a preset driving trajectory algorithm.
[0116] For example, a specific time may be set to N seconds from the present (where N is a real number greater than 0). For example, the driving trajectory prediction unit may generate a driving trajectory that is expected to be driven by a railway vehicle for N seconds from the present. As another example, the specific time may be set to the time when the railway vehicle is expected to drive on a specific track based on track location information. However, the specific time may be set in various ways, not limited to the present embodiment.
[0117] As another example, as previously mentioned, the track location information may correspond to the three-dimensional location of the track existing in the direction in which the railway vehicle is traveling. In this embodiment, it may correspond to the current location of the track existing in the direction in which the railway vehicle is currently traveling. Additionally, as previously mentioned, the railway vehicle location information corresponds to the location of the railway vehicle, and in this embodiment, it may correspond to the current location of the railway vehicle currently traveling. Additionally, as previously mentioned, the railway vehicle speed information corresponds to the speed of the railway vehicle, and in this embodiment, it may correspond to the current driving speed of the railway vehicle currently traveling. Additionally, as previously mentioned, the railway vehicle path information may correspond to the expected driving path of the railway vehicle.
[0118] In this case, the pre-set driving trajectory algorithm may be an LSTM algorithm or a GAN (Generative Adversarial Network) algorithm.
[0119] However, the present embodiment is not limited to this, and the predicted driving trajectory can be generated through various methods, algorithms, and models capable of generating the predicted driving trajectory.
[0120] Meanwhile, the object trajectory prediction step can generate an object predicted trajectory, which is a path expected to be traveled by the object during a specific period of time, based on the object's trajectory and the railway vehicle's driving information. (S640)
[0121] For example, the object trajectory prediction step can generate an object predicted trajectory by inputting the object's trajectory and the railway vehicle's driving information into a preset object trajectory algorithm.
[0122] For example, in the same way as the driving trajectory prediction step, an object's predicted trajectory for a specific period of time can be generated.
[0123] The embodiments of the present disclosure relate to a technology for preventing a railway vehicle from colliding with an object in advance. Accordingly, the collision risk determination unit described below determines whether there is a risk of collision by comparing the predicted trajectory of the object with the predicted trajectory of the vehicle. The determination of whether there is a risk of collision needs to be based on the same time period. Accordingly, the object trajectory prediction unit also needs to generate the predicted trajectory of the object for a specific time period in the same way as the vehicle trajectory prediction step.
[0124] As another example, the railway vehicle speed information included in the railway vehicle driving information and the trajectory of the object generated in the object trajectory calculation step can be input into an algorithm that is pre-set as input values.
[0125] In this case, the pre-set object trajectory algorithm may include deep learning algorithms such as LSTM and GRU, and reinforcement learning algorithms such as DDPG (Deep Deterministic Policy Gradient) which can predict continuous state values.
[0126] In addition, the preset algorithm may be an algorithm combined with either a Kalman filter or an extended Kalman filter used in the object trajectory calculation step, and either a Long Short-Term Memory (LSTM) or a Gated Recurrent Unit (GRU).
[0127] However, the present embodiment is not limited to this embodiment, and the object trajectory prediction step may utilize various methods, algorithms, and models as long as they can generate an object predicted trajectory.
[0128] Meanwhile, the collision risk determination step can determine the risk of collision between the railway vehicle and the object by comparing the expected driving trajectory and the expected object trajectory. (S650)
[0129] For example, the collision risk determination step may determine that there is a risk of collision with the object if an overlapping section occurs between the expected driving trajectory and the expected object trajectory.
[0130] There is a point where the object's expected trajectory and the railway vehicle's expected trajectory overlap. In this case, the collision risk judgment step can determine that there is a risk of collision with the object.
[0131] However, the risk of collision with an object can be determined in various ways, not limited to the present embodiment.
[0132] Meanwhile, the control step can control the railway vehicle to perform a warning action based on the result of the collision risk judgment step. (S660)
[0133] For example, a warning action of a railway vehicle may include one or more of the following: an action of displaying a warning message to the railway vehicle, an action of decelerating the railway vehicle, and an action of emergency braking the railway vehicle.
[0134] For example, a warning message on a railway vehicle may be displayed through an output device of the railway vehicle. In this case, the output device may include a display, an LED, and a speaker, and the output device may be installed inside or outside the railway vehicle.
[0135] As another example, if it is determined that there is a risk of collision with an object, the deceleration operation of a railway vehicle can be controlled by transmitting a control command to the vehicle's driving device in the control stage to perform the deceleration operation of the railway vehicle.
[0136] As another example, if it is determined that there is a risk of collision with an object, the control stage can transmit a control command to the vehicle's driving device to control the railway vehicle to perform an emergency braking action.
[0137] The above-described embodiments may be implemented within a computer system, for example, on a computer-readable recording medium. The computer system of a railway vehicle control device may include at least one element among one or more processors, memory, storage, user interface inputs, and user interface outputs, and these may communicate with each other via a bus. Additionally, the computer system may also include a network interface for connecting to a network. The processor may be a CPU or a semiconductor device that executes processing instructions stored in memory and / or storage. Memory and storage may include various types of volatile / non-volatile memory media. For example, memory may include ROM and RAM.
[0138] The foregoing description is merely an illustrative explanation of the technical concept of the present disclosure, and those skilled in the art to which the present disclosure pertains may make various modifications and variations within the scope of the essential characteristics of the technical concept. Furthermore, since these embodiments are intended to explain, not limit, the scope of the technical concept is not limited by these embodiments. The scope of protection of the present disclosure shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present disclosure.
[0139]
[0140] CROSS-REFERENCE TO RELATED APPLICATION
[0141] This patent application claims priority pursuant to Section 119(a) of the U.S. Patent Act (35 USC § 119(a)) to Korean Patent Application No. 10-2024-0183331 filed on December 11, 2024, the entire contents of which are incorporated by reference into this patent application. Furthermore, this patent application claims priority in countries other than the United States for the same reasons as above, and the entire contents of which are incorporated by reference into this patent application.
Claims
1. A receiver that receives image data, LiDAR data, and driving information of the railway vehicle using one or more sensors of the railway vehicle; An object trajectory calculation unit that recognizes an object and a track area based on the above image data and LiDAR data, and generates a trajectory of the object which is the path the object traveled; A driving trajectory prediction unit that generates a driving trajectory, which is a driving trajectory expected to be driven by the railway vehicle for a specific period of time, based on the above track area and driving information of the above railway vehicle; An object trajectory prediction unit that generates an object predicted trajectory, which is a path expected to be traveled by the object during the specific time period, based on the trajectory of the object and the driving information of the railway vehicle; A collision risk determination unit that determines the risk of collision between the railway vehicle and the object by comparing the above-mentioned predicted driving trajectory and the above-mentioned predicted object trajectory; and A railway vehicle control device comprising a control unit that controls the railway vehicle to perform a warning operation according to the judgment result of the collision risk judgment unit.
2. In Paragraph 1, The above object trajectory calculation unit is, Extract image object feature data from the above image data, and Extract LiDAR object feature data from the above LiDAR data, and A railway vehicle control device that recognizes the object and track area using combined data generated by combining the image object feature data and LiDAR object feature data.
3. In Paragraph 2, The above object trajectory calculation unit is, A railway vehicle control device that generates the trajectory of an object using object position information and object velocity information included in the combined data.
4. In Paragraph 1, The above driving trajectory prediction unit is, Track location information included in the above track area, and A railway vehicle control device that generates the predicted driving trajectory by inputting railway vehicle location information, railway vehicle speed information, and railway vehicle route information included in the driving information of the above railway vehicle into a preset driving trajectory algorithm.
5. In Paragraph 1, The above object trajectory prediction unit is, A railway vehicle control device that generates an object predicted trajectory by inputting the trajectory of the object and railway vehicle speed information included in the driving information of the railway vehicle into a preset object trajectory algorithm.
6. In Paragraph 1, The above collision risk determination unit is, A railway vehicle control device that determines there is a risk of collision with an object when an overlapping point occurs between the above-mentioned predicted driving trajectory and the object's predicted trajectory.
7. In Paragraph 6, The above control unit is, A railway vehicle control device that controls the railway vehicle to perform a warning operation when it is determined that there is a risk of collision with the above object.
8. In Paragraph 1, The warning operation of the above railway vehicle is, A railway vehicle control device comprising one or more of the following operations: displaying a warning message to the railway vehicle, decelerating the railway vehicle, and emergency braking the railway vehicle.
9. In Paragraph 1, One or more sensors of the above railway vehicle are It may include one or more of a front camera, a rear camera, a right camera, and a left camera capable of receiving the above image data, and A railway vehicle control device capable of receiving the above-mentioned lidar data, comprising one or more of a front lidar, a rear lidar, a right lidar, and a left lidar.
10. A receiving step of receiving image data, LiDAR data, and driving information of the railway vehicle using one or more sensors of the railway vehicle; An object trajectory calculation step that recognizes an object and a track area based on the above image data and LiDAR data, and generates the trajectory of the object, which is the path the object moved along; A driving trajectory prediction step for generating a driving trajectory, which is a driving trajectory expected to be driven by the railway vehicle for a specific period of time, based on the track area and driving information of the railway vehicle; An object trajectory prediction step for generating an object predicted trajectory, which is a path expected to be traveled by the object during the specific time period, based on the trajectory of the object and the driving information of the railway vehicle; A collision risk determination step for determining the risk of collision between the railway vehicle and the object by comparing the above-mentioned predicted driving trajectory and the above-mentioned predicted object trajectory; and A railway vehicle control method comprising a control step for controlling the railway vehicle to perform a warning operation according to the result of the collision risk judgment step.
11. In Paragraph 10, The above object trajectory calculation step is, Extract image object feature data from the above image data, and Extract LiDAR object feature data from the above LiDAR data, and A railway vehicle control method for recognizing the object and track area using combined data generated by combining the image object feature data and LiDAR object feature data.
12. In Paragraph 11, The above object trajectory calculation step is, A railway vehicle control method for generating a trajectory of an object using object location information and object velocity information included in the combined data.
13. In Paragraph 10, The above driving trajectory prediction step is, Track location information included in the above track area, and A railway vehicle control method for generating a predicted driving trajectory by inputting railway vehicle location information, railway vehicle speed information, and railway vehicle path information included in the driving information of the above railway vehicle into a preset driving trajectory algorithm.
14. In Paragraph 10, The above object trajectory prediction step is, A railway vehicle control method for generating an object predicted trajectory by inputting the trajectory of the object and railway vehicle speed information included in the driving information of the railway vehicle into a preset object trajectory algorithm.
15. In Paragraph 10, The above collision risk determination step is, A railway vehicle control method that determines there is a risk of collision with an object when an overlapping point occurs between the predicted driving trajectory and the predicted object trajectory.
16. In Paragraph 15, The above control step is, A railway vehicle control method that controls the railway vehicle to perform a warning action when it is determined that there is a risk of collision with the above object.
17. In Paragraph 10, The warning operation of the above railway vehicle is, A railway vehicle control method comprising one or more of the following: displaying a warning message to the railway vehicle, decelerating the railway vehicle, and emergency braking the railway vehicle.
18. In Paragraph 10, One or more sensors of the above railway vehicle are It may include one or more of a front camera, a rear camera, a right camera, and a left camera capable of receiving the above image data, and A railway vehicle control method that may include one or more of a front lidar, a rear lidar, a right lidar, and a left lidar capable of receiving the above lidar data.