Apparatus and method for identifying objects near a railway
By generating a depth map from monocular images using rail positions as a reference, the method effectively filters out irrelevant objects, enhancing the accuracy and efficiency of identifying encroaching objects on railways.
Patent Information
- Application Number
- PCT/GB2026/050779
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-05-08
- Publication Date
- 2026-11-12
AI Technical Summary
Existing methods for identifying objects encroaching on railways using monocular images are unreliable and inaccurate due to the lack of consideration of depth information, leading to inefficient and ineffective object detection, particularly with vegetation.
Create a depth map from a monocular image to filter out irrelevant or distant objects by using the known distance between rails as a reference, allowing for accurate identification of objects of interest within a defined area of interest.
Improves the accuracy and efficiency of identifying objects encroaching on railways by filtering depth maps based on rail positions, simplifying the identification process and reducing false positives.
Smart Images

Figure GB2026050779_12112026_PF_FP_ABST
Abstract
Description
[0001] APPARATUS AND METHOD FOR IDENTIFYING OBJECTS NEAR A RAILWAY
[0002] TECHNOLOGICAL FIELD
[0003] Various example embodiments relate to an apparatus and a method for identifying objects of interest near a railway.
[0004] BACKGROUND
[0005] Railways need to be kept clear of obstructions so that trains may travel safely along the railway. Objects, such as vegetation, which may encroach on a railway in a dangerous manner need to be identified so that any potential safety issues can be addressed.
[0006] Existing techniques to monitor the encroachment of objects on a railway includes the use of LIDAR which uses laser beams to measure the distances to objects. LIDAR systems mounted on a train have successfully been used to create a 3D model of the environment around the train on a railway. In this way, any objects encroaching onto the railway can be identified. Similarly, stereoscopic cameras for capturing 3D images have been used to help detect objects encroaching onto the railway. However, both of these methods require expensive specialist imaging systems to be mounted to the train. It would be desirable to detect objects encroaching onto the railway without the need for expensive specialist imaging systems.
[0007] Monocular images may be captured using cheaper and more readily available cameras. It has been attempted to identify encroaching objects on a railway using images captured from such cameras. However, results so far have not been reliable. One method involves using multi-frame image processing techniques such as mono vslam (Monocular Visual Simultaneous Localisation and Mapping) which uses a plurality of video frames to calculate the position and orientation of a camera with respect to its surroundings while simultaneously mapping the 3D structure of the environment. However, such multi-frame image processing techniques often struggle to identify objects or features repeatably across adjacent frames in highly compressed and motion blurred video footage, resulting in poor mapping estimates. This is especially the case when trying to identify vegetation, which often passes quickly at the edges of the frame. Alternatively, single frame machine learning based techniques have been developed which utilise methods such as object detection, classification and segmentation to identify possible encroachment. However, the identification of objects of interest encroaching onto the railway may be inaccurate.It would be desirable to provide a technique for identifying objects of interest which encroach on a railway which does not suffer from the same drawbacks as the existing methods.
[0008] BRIEF SUMMARY
[0009] The scope of protection sought for various example embodiments of the invention is set out by the independent claims. The example embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.
[0010] According to a first aspect of the invention, there is provided an apparatus comprising: at least one processor; and at least one memory storing instructions that when executed by the at least one processor cause the apparatus at least to perform: obtaining a monocular image of a railway; identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; selecting a reference point within the monocular image based on the position of the rails; calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
[0011] Objects of interest, including vegetation, which encroach on a railway need to be identified such that railways can be kept clear and safe. Cameras, often mounted to the front of a train, may capture video clips or images of a railway. The present invention relates to a process for identifying objects of interest within an image which encroach on the railway.
[0012] Compared to LIDAR systems and stereoscopic cameras, the present invention may identify objects of interest within a monocular image. Therefore, the invention can avoid the need to mount expensive specialist imaging systems to the train.
[0013] It was recognised that existing methods for identifying objects of interest within monocular images (e.g., multi-frame image processing techniques and single frame machine learning based techniques) may be unreliable or inaccurate because they do not consider the depth of the objects in the image. Embodiments of the inventiontherefore create a depth map from the monocular image such that depth information can be considered when identifying objects of interest.
[0014] In summary, the invention identifies the position of the rails within the monocular image. Since the rails are a known distance apart, this allows distances within the image to be calculated. Furthermore, a depth map is created from the monocular image using any known technique or publicly available tool. A reference point is selected in the image based on the identified rails and a baseline depth can be obtained by finding the equivalent point in the depth map. The depth map is filtered for objects which are at or closer than the baseline depth. Objects of interest are then identified within the monocular image based on the filtered depth map. Specifically, the filtered depth map may be used to identify objects which encroach on the railway and these objects in the equivalent portions of the monocular image may be analysed to identify if they are objects of interest. The quality of the identification of objects of interest may be improved compared to previous attempts at single frame I monocular image object identification techniques because information that is not relevant (e.g., because it does not encroach on the railway) or not useable (e.g., because it is too far away from the camera) may be removed by the depth map filtering, thereby simplifying the identification task. Moreover, basing the depth map filtering on the identified position of the rails may help select a consistent depth at which to filter and ensure that only relevant information is considered when identifying objects of interest. For example, portions of the image which are less useful (e.g., portions within the image which are far away from the camera or are not within or directly adjacent to the path of the train) may be ignored.
[0015] In some embodiments, the apparatus is further caused to perform: determining, based on the position of the rails, an area of interest within the monocular image in a plane of the baseline depth; and masking the depth map using the area of interest such that information outside of the area of interest is discarded, wherein the filtering the depth map comprises filtering the masked depth map.
[0016] The area of interest (which may be referred to as an envelope) is an area in which objects are desired to be identified within. For example, the area of interest may comprise the path of the train and an area immediately adjacent the path of the train.
[0017] By discarding all information outside of the area of interest and subsequently filtering out any objects which are beyond the baseline depth, the task of identifying any objectsof interest can be simplified and may become more efficient. This may lead to a faster and more accurate identification of objects of interest.
[0018] Note that the size of the area of interest within the image may be determined based on the position of the rails because the rails are a known distance apart.
[0019] In some embodiments, the area of interest is a predetermined size and / or shape.
[0020] As discussed above, the area of interest is an area in which objects are desired to be identified within. The dimensions of the area of interest may be standardised. In other words, standards may define the size of a channel on the railways which should be kept clear for the trains. For example, standards may require a 4x6m rectangular channel centred on the rails to be kept clear of vegetation. Therefore, in some embodiments, the area of interest within the image is determined to be the standardised area or based on the standardised area (e.g., the standardised area plus / minus a margin).
[0021] In some embodiments, the reference point is selected such that the area of interest fits within the monocular image at the baseline depth.
[0022] The effective field of view for objects in an image increases with increasing distance from the camera. In other words, the camera captures a wider physical area at a distance and a narrower physical area closer to the camera. This means that objects which are close to the camera but to the side of the train may not be shown in the image.
[0023] If the reference point is selected to be too close to the train (i.e., in the foreground of the image), the effective field of view at that distance is limited and may not show the whole area of interest (which as discussed above may have a predetermined size). As a result, objects of interest within the area of interest may not be identified.
[0024] In this embodiment, the reference point is selected such that the entire area of interest is contained within the image at the baseline depth to help ensure potentially important information is not lost due to the reference point being too close to the camera.In some embodiments, the reference point is selected based on a quality of the image at the baseline depth. In some embodiments, the reference point is selected such that a quality of the image at the baseline depth meets a threshold requirement.
[0025] Objects that are far away from the camera (i.e. , in the background of the image) are often captured with worse quality compared to objects that are closer to the camera. In one embodiment, the quality of the image at the depth of the reference point is checked against a threshold requirement to ensure that the quality of information (e.g., the detail of the objects) in the image at the depth of the reference point is sufficient to identify the objects of interest to a desired degree of reliability and accuracy. If the quality of the portion of the image in which objects are being identified is too low due to the reference point being too far away from the camera, the identification of the objects of interest may become more difficult and less accurate.
[0026] Image quality may be measured using any known technique. For example, a blur of objects at a particular distance may be measured to determine an appropriate distance at which to set the reference point to ensure sufficient image quality for the objects of interest to be identified.
[0027] It will be appreciated that a balancing act may be performed between fitting the area of interest in the image at the baseline depth and not having a baseline depth too far away such that the identification of objects of interest degrades due to the reduced image quality.
[0028] It will be appreciated that the reference point may be selected based on the depth map because information about the depth of objects within the monocular image may be considered when selecting the reference point.
[0029] In some embodiments, the nature of the objects of interest is taken into account when selecting the reference point. For example, if the target is to identify high up or overhanging vegetation, then the reference point should be selected to be far enough away from the camera so that the target vegetation is not filtered out or discarded.
[0030] In some embodiments, the reference point is selected based on calibration information relating to a camera used to obtain the monocular image.The calibration may include, but is not limited to, the camera field of view, the angle of orientation of the camera and the focal length of the camera. It will be appreciated that the sweet spot for the reference point may vary based on this calibration information. The calibration information may be obtained by retrieving it from a database or receiving it from another apparatus. The calibration information may be predetermined and known to the apparatus.
[0031] In some embodiments, selecting the reference point comprises receiving an indication of the reference point from an input by a user. In this way, a user may manually select the reference point in the image. The user may consider the factors discussed above when selecting the reference point. For example, the user may make a manual assessment of the quality of objects captured at different depths and select an appropriate reference point.
[0032] In some embodiments, the identifying the position of the rails comprises using a first machine learning algorithm trained to identify rails in images.
[0033] In some embodiments, the monocular image is captured from a train travelling along the rails of the railway and the monocular image contains a plurality of sets of rails, the identifying the position of the rails comprises identifying the rails on which the train is travelling and ignoring any sets of rails on which the train is not travelling. In this way, additional sets of rails which should not be considered for the identification of objects encroaching on the railway can be identified and ignored such that the incorrect identification of encroaching objects may be avoided.
[0034] In some embodiments, the identifying the objects of interest comprises using at least one of the following: object detection; contour detection; colour filtering; local binary patterns; image classification; and semantic segmentation.
[0035] In some embodiments, the identifying the objects of interest comprises using a second machine learning algorithm trained to identify the objects of interest.
[0036] In some embodiments, the objects of interest comprise vegetation.
[0037] To maintain a safe railway, vegetation encroaching on the railway must be identified and removed when it grows too close to or into the path of the train.In some embodiments, the apparatus is further caused to perform: calculating a coverage of the identified objects of interest within the monocular image; and generating a first indication indicative of the coverage.
[0038] The indication may comprise a percentage of the image (or area of interest) which is covered by the identified objects of interest. For example, the percentage coverage of the area of interest of vegetation may be calculated. Alternatively or additionally, the indication may comprise a score indicative of the level of coverage. The indication may be provided to or displayed to a user or operator of the apparatus. In some embodiments, an additional indication may be generated if the coverage exceeds a threshold amount. The threshold amount may represent a level of coverage which is not tolerable. In this way, an alert may be generated whenever action is required to reduce the encroachment on the railway by the objects of interest.
[0039] In some embodiments, the first indication is weighted based on a proximity of the identified objects of interest to the position of the rails.
[0040] In this way, the indication (e.g., a score indicative of the level of coverage) may be more reactive to the encroachment of the objects of interest on the railway becoming more severe.
[0041] In some embodiments, the monocular image comprises a plurality of monocular images.
[0042] In other words, the process may be performed on multiple images. In some embodiments, the plurality of monocular images is a continuous sequence of monocular images from a video clip.
[0043] Processing multiple images with a similar view may help reduce the uncertainty in the identification of the objects of interest at certain points along the railway. In some embodiments, the results from multiple images may be condensed to make them more appropriate for human consumption. For example, a single coverage score may be generated which indicates a rolling average from a continuous sequence of monocular images.
[0044] In some embodiments, the apparatus is further caused to perform: determining whether the coverage for each of the plurality of monocular images meets a threshold coverage;and if a number of monocular images which meets the threshold coverage meets a threshold number, generating a second indication indicative of the presence of the objects of interest.
[0045] When the coverage is significant in multiple images, this may be a strong indicator that objects of interest are encroaching on the railway. Therefore, when a certain number of images or percentage of images from the plurality of images reaches a threshold, an indication of the potentially encroaching objects of interest is generated. In some embodiments, the second indication may be generated when a rolling average coverage over successive image frames exceeds a configurable threshold. The indication may be provided to or displayed to the user.
[0046] In some embodiments, the indication may be a metric which allows the user to ascertain the level of encroachment along a route (i.e. , continuous rather than discrete). The indication may be based on a rolling average coverage. In this way, the apparatus may provide an indication when the level of encroachment is high, but also when there is no encroachment or the encroachment is within acceptable levels.
[0047] According to a second aspect of the invention, there is provided a system comprising: an apparatus according to the first aspect; and a camera for capturing the monocular image. The camera may be mounted on a train travelling along the railway. The camera may not be able to capture stereoscopic images and / or videos.
[0048] According to a third aspect of the invention, there is provided an apparatus comprising: means for obtaining a monocular image of a railway; means for identifying a position of rails of the railway within the monocular image; means for generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; means for selecting a reference point within the monocular image based on the position of the rails; means for calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; means for filtering the depth map for objects at or closer than the baseline depth; and means for identifying objects of interest within the monocular image based on the filtered depth map.
[0049] The means may perform the optional features set out in relation to the apparatus mentioned above.The processor, memory, and example algorithms, encoded as instructions, program, or code, may be the means for providing or causing the performance of the operation.
[0050] According to a fourth aspect of the invention, there is provided an apparatus comprising: circuitry configured for obtaining a monocular image of a railway; circuitry configured for identifying a position of rails of the railway within the monocular image; circuitry configured for generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; circuitry configured for selecting a reference point within the monocular image based on the position of the rails; circuitry configured for calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; circuitry configured for filtering the depth map for objects at or closer than the baseline depth; and circuitry configured for identifying objects of interest within the monocular image based on the filtered depth map.
[0051] The circuitry may be configured perform the optional features set out in relation to the apparatus mentioned above.
[0052] According to a fifth aspect of the invention, there is provided a method comprising: obtaining a monocular image of a railway; identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; selecting a reference point within the monocular image based on the position of the rails; calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
[0053] In some embodiments, the method further comprises: determining, based on the position of the rails, an area of interest within the monocular image in a plane of the baseline depth; and masking the depth map using the area of interest such that information outside of the area of interest is discarded, wherein the filtering the depth map comprises filtering the masked depth map.
[0054] In some embodiments, the area of interest is a predetermined size and / or shape.In some embodiments, the reference point is selected such that the area of interest fits within the monocular image at the baseline depth.
[0055] In some embodiments, the reference point is selected based on a quality of the image at the baseline depth. In some embodiments, the reference point is selected such that a quality of the image at the baseline depth meets a threshold requirement.
[0056] In some embodiments, the reference point is selected based on calibration information relating to a camera used to obtain the monocular image.
[0057] In some embodiments, selecting the reference point comprises receiving an indication of the reference point from an input by a user.
[0058] In some embodiments, the identifying the position of the rails comprises using a first machine learning algorithm trained to identify rails in images.
[0059] In some embodiments, the monocular image is captured from a train travelling along the rails of the railway and the monocular image contains a plurality of sets of rails, the identifying the position of the rails comprises identifying the rails on which the train is travelling and ignoring any sets of rails on which the train is not travelling.
[0060] In some embodiments, the identifying the objects of interest comprises using at least one of the following: object detection; contour detection; colour filtering; local binary patterns; image classification; and semantic segmentation.
[0061] In some embodiments, the identifying the objects of interest comprises using a second machine learning algorithm trained to identify the objects of interest.
[0062] In some embodiments, the objects of interest comprise vegetation.
[0063] In some embodiments, the method further comprises: calculating a coverage of the identified objects of interest within the monocular image; and generating a first indication indicative of the coverage.
[0064] In some embodiments, the first indication is weighted based on a proximity of the identified objects of interest to the position of the rails.In some embodiments, the monocular image comprises a plurality of monocular images.
[0065] In some embodiments, the method further comprises: determining whether the coverage for each of the plurality of monocular images meets a threshold coverage; and if a number of monocular images which meets the threshold coverage meets a threshold number, generating a second indication indicative of the presence of the objects of interest.
[0066] According to a sixth aspect of the invention, there is provided a computer program comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: obtaining a monocular image of a railway; identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; selecting a reference point within the monocular image based on the position of the rails; calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
[0067] The instructions may be for performing the optional features set out in relation to the method mentioned above.
[0068] According to a seventh aspect of the invention, there is provided a computer program comprising instructions stored thereon for performing at least the following: obtaining a monocular image of a railway; identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; selecting a reference point within the monocular image based on the position of the rails; calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
[0069] The instructions may be for performing the optional features set out in relation to the method mentioned above.According to an eighth aspect of the invention, there is provided a non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following: obtaining a monocular image of a railway; identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image; selecting a reference point within the monocular image based on the position of the rails; calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image; filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
[0070] The instructions may be for performing the optional features set out in relation to the method mentioned above.
[0071] Further particular and preferred aspects are set out in the accompanying independent and dependent claims. Features of the dependent claims may be combined with features of the independent claims as appropriate, and in combinations other than those explicitly set out in the claims.
[0072] Where an apparatus feature is described as being operable to provide a function, it will be appreciated that this includes an apparatus feature which provides that function or which is adapted or configured to provide that function.
[0073] BRIEF DESCRIPTION
[0074] Some example embodiments will now be described with reference to the accompanying drawings in which:
[0075] Figure 1 illustrates a monocular image being processed according to an embodiment of the invention;
[0076] Figure 2 illustrates a depth map of the monocular image of Figure 1;
[0077] Figure 3 illustrates a depth map of the monocular image of Figure 1;
[0078] Figure 4 illustrates vegetation of interest identified within the monocular image of Figure 1;
[0079] Figure 5 illustrates an object identified within the monocular image of Figure 1;
[0080] Figure 6 illustrates the results of the process according to an embodiment of the invention overlaid on the monocular image of Figure 1;
[0081] Figure 7 illustrates the steps in a method according to an embodiment of the invention; andFigure 8 illustrates an apparatus according to an embodiment of the invention.
[0082] DETAILED DESCRIPTION
[0083] Before discussing the example embodiments in any more detail, first an overview will be provided.
[0084] Objects of interest, including vegetation, which encroach on a railway need to be identified such that railways can be kept clear and safe. Cameras mounted to a train may capture video clips or images of a railway. Embodiments relate to a process for identifying objects of interest which encroach on a railway.
[0085] LIDAR uses laser beams to measure distances and create a 3D model. LIDAR systems mounted on a train have successfully been used to create a 3D model of the environment around the train on a railway. In this way, any objects encroaching onto the railway can be identified. Similarly, stereoscopic cameras for capturing 3D images have been used to help detect objects encroaching onto the railway. However, both these methods require expensive specialist imaging systems to be mounted to the train. Embodiments aim to detect objects encroaching onto the railway without the need for expensive specialist imaging systems.
[0086] Monocular images may be captured using cheaper and more common cameras.
[0087] Images captured from such cameras may be used to identify encroaching objects. One method involves using multi-frame image processing techniques such as mono vslam (Monocular Visual Simultaneous Localisation and Mapping) which uses a plurality of video frames to calculate the position and orientation of a camera with respect to its surroundings while simultaneously mapping the environment. However, such multiframe image processing techniques often struggle to identify objects on highly compressed and motion blurred video footage, especially when trying to identify vegetation which often passes quickly at the edges of the frame. Alternatively, single frame machine learning based techniques have been explored which utilise methods such as object detection, classification and segmentation. However, the identification of objects of interest encroaching onto the railway may be inaccurate. Embodiments aim to provide a technique for identifying objects of interest within a monocular image without suffering from the same drawbacks as the existing techniques.
[0088] Embodiments may identify objects of interest (i.e. , objects encroaching onto the railway) using a monocular image of the railway. The proposed technique involvesidentifying the position of the rails in the image. Since the rails are a known distance apart, this allows distances within the image to be calculated. The proposed technique further includes creating a depth map from the monocular image using any known technique or publicly available tool. A reference point is selected in the image based on the identified rails and a baseline depth can be obtained by finding the equivalent point in the depth map. The depth map is filtered for objects which are at or closer than the baseline depth. Objects of interest are then identified within the filtered image. The quality of object identification may be improved compared to previous attempts at single frame object identification because extraneous detail may be removed using the depth map filtering, thereby simplifying the identification task.
[0089] In some embodiments, the proposed technique includes determining, based on the position of the rails, an area of interest within the monocular image in a plane of the baseline depth and masking the depth map using the area of interest such that information outside of the area of interest is discarded. Once more, extraneous detail may be removed from the image to further simplify the identification of objects of interest.
[0090] In some embodiments, the area of interest is a predetermined size and / or shape. For example, the area of interest may encompass the path of the train plus a distance from the train within which objects are desired to be identified. This distance may be standardised. For example, standards may require vegetation to be kept a minimum distance (e.g., 1.5m) away from the train.
[0091] Figure 1 shows a monocular image 10 of a railway being processed according to an embodiment of the invention. The embodiment provides a method for identifying objects of interest, such as vegetation, in the monocular image 10.
[0092] The monocular image 10 was captured by a camera mounted to the front of a train travelling along the rails 12. The monocular image 10 may be one of a plurality of images from a video captured by the camera. Calibration information such as the position and angle of the camera relative to the rails and the field of view of the camera may or may not be known. It will be appreciated that the monocular image 10 of the railway may be obtained in any suitable way. For example, the image may be retrieved from a datastore.According to the embodiment, the position of the rails 12 within the monocular image 10 is identified. The identified position is shown in Figure 1 by the lines overlaid on the monocular image 10 on top of the rails 12. In this embodiment, the position of the rails is taken to be the railheads. Identifying the position of the rails 12 in the frame allows a baseline depth to be selected (as discussed in more detail below) and can provide a real-world reference point for dimensions within the monocular image 10 because the rails 12 are a known fixed distance apart.
[0093] In some embodiments, the position of the rails may be identified using a first machine learning algorithm trained to identify rails in images. The machine learning algorithm may be trained using frames from video footage captured by a camera mounted to a train. In other embodiments, the position of the rails may be manually identified.
[0094] However, a manual identification of the rails may be less accurate and / or less consistent if performing the process on multiple images.
[0095] In some embodiments, the monocular image 10 contains a plurality of sets of rails. In this case, identifying the position of the rails comprises identifying the rails 12 on which the train to which the camera is mounted is travelling and ignoring any sets of rails on which the train is not travelling. For example, a combination of machine learning and computer vision techniques may be used to first segment the monocular image and then a directed acyclic edge graph may be used to prune non-running rails.
[0096] The process further includes generating a depth map 20 of the monocular image 10. The depth map 20 indicates the distance to surfaces of objects within the monocular image 10. It will be appreciated that the depth map 20 may be created before, at the same time as or after the step of identifying the position of the rails 12.
[0097] Figure 2 shows an example of a depth map 20 of the monocular image 10 which has been created and normalized such that a ‘relative’ map is produced. Publicly available depth estimation tools may be used to convert the monocular image 10 into an estimated depth map 20.
[0098] Figure 3 shows a depth map 30 of the monocular image 10 which uses an alternative false colour scheme to help highlight points in the image which are close to the camera and points which are far from the camera. The depth map 20 or the depth map 30 may be displayed to a user.Based on the identified position of the rails 12, a reference point 14 within the monocular image is selected. In this embodiment, the reference point 14 is selected to be a point along one of the rails 12. In particular, the reference point 14 is selected to be a point which is a fixed number of pixels from the bottom of the image 10. The number of pixels may depend on the calibration information of the camera.
[0099] A baseline depth is then calculated based on a point in the depth map 20 which corresponds to the reference point 14 within the monocular image 10.
[0100] In some embodiments, the reference point 14 is selected at a given height in the frame based on the detected running rails 12 and based on the camera’s calibration if available. For example, to select the reference point (sometimes referred to as the baseline point):
[0101] • the first step is to determine the distance along the rails from the camera (referred to as the baseline Y coordinate):
[0102] o where the camera calibration is known, a predetermined value can be used, e.g., for a given camera, this could be 400 pixels from the bottom of the frame;
[0103] o otherwise, the distance can be calculated by finding the Y coordinate of the image where the distance between the rails is a target number of pixels;
[0104] • given the baseline Y coordinate and the detected rails, the next step is to find the X coordinate of each rail at that Y;
[0105] • given the X,Y coordinate of the rail and depth map, find the depth at the railhead for each point may be found; and
[0106] • given the depths at each rail, whichever point has the smaller depth becomes the reference point.
[0107] It will be appreciated that which rail the reference point 14 is positioned on may alternatively be determined by which is furthest from the camera (i.e. , has the greatest depth). In some embodiments, the reference point may be a mid-point between the two rails. However, the depth from the camera at this location may not be consistent due to the sleepers and track ballast. Therefore, selecting the reference point 14 to be along the railhead of the rails 12 may have less noise.In some embodiments, the reference point is selected such that a quality of the image at the baseline depth meets a threshold requirement. For example, objects that are far away from the camera may be captured with worse quality compared to objects that are closer to the camera. The quality of the image at the depth of the reference point may be checked against a threshold requirement to ensure that the quality of information (e.g., the detail of the objects) in the image at the depth of the reference point is sufficient to identify the objects of interest to a desired degree of reliability and accuracy. The image quality may be measured using any known technique. For example, a blur of objects at a particular distance may be measured to determine an appropriate distance at which to set the reference point to ensure sufficient image quality for the objects of interest to be identified.
[0108] When selecting the reference point 14 does not depend on depth information from the depth map 20, it will be appreciated that selecting the reference point 14 may be performed before, at the same time as or after the step of generating the depth map 20.
[0109] In some embodiments, the process further includes determining, based on the position of the rails 12, an area of interest 16 (sometimes referred to as an envelope) within the monocular image 10 in a plane of the baseline depth as shown in Figure 1. The area of interest 16 is an area within the monocular image 10 within which it is desired to identify the objects of interest. In other words, the area of interest 16 is defined in a plane within the monocular image 10 in which encroachment will be measured. The area of interest 16 is parallel to the front of the train but at the baseline depth. In the embodiment shown in Figure 1, the area of interest is determined to be raised slightly from the reference point 14 on the rails 12. It was recognized that there may be many objects which are not of interest on the ground or at a similar level to the rails. These objects may lead to false positives when identifying objects of interest. By raising the area of interest above the rails (but still in the plane of the reference point 14), such noise can be ignored and the number of false positives reduced. The area of interest may be raised a fixed number of pixels above the reference point 14 or a real world distance determined based on the distance between the identified rails 12.
[0110] In some embodiments, the area of interest 16 is a predetermined size and / or shape. For example, the area of interest may comprise the path of the train and an area immediately adjacent the path of the train. The dimensions of the area of interest may be standardised. For example, standards may require a rectangular channel centred onthe rails to be kept clear of vegetation. Therefore, in some embodiments, the area of interest 16 within the image is determined to be the standardised area or the standardised size plus / minus a margin. The area of interest 16 may be projected in the baseline depth using a multiplier of the known distance between the rails in order to meet a real world target span. In the vertical direction, the area of interest may be extended to the top of the image frame or to a fixed height.
[0111] In some embodiments, the reference point is selected such that the area of interest 16 fits within the monocular image at the baseline depth. The effective field of view in an image increases with increasing distance from the camera. In other words, the camera captures a wider physical area at a greater distance and a narrower physical area closer to the camera. This means that objects which are close to the camera but to the side of the train may not be shown in the image. If the reference point is selected to be too close to the train (i.e., in the foreground of the image), the effective field of view at this distance may be limited and may not show the whole area of interest (which may have a predetermined size). As a result, objects of interest within the area of interest may not be identified. Therefore, in this embodiment, the reference point is selected such that the entire area of interest is contained within the image at the baseline depth to help ensure potentially important information is not lost due to the reference point being too close to the camera. It will be appreciated that a balancing act may be performed between fitting the area of interest in the image at the baseline depth and not having a baseline depth too far away such that the identification of objects of interest degrades due to the reduced image quality.
[0112] Once determined, the area of interest 16 is used to mask the depth map 20 such that information outside of the area of interest 16 is discarded.
[0113] Subsequently, the masked depth map 20 is filtered for all points / objects at or closer than the baseline depth. Consequently, all information outside the area of interest 16 and beyond the baseline depth is discarded and only information within the area of interest 16 and within a desired depth remains (this may be referred to as the encroachment mask). By discarding all information outside of the area of interest and subsequently filtering out any objects which are beyond the baseline depth, the task of identifying any objects of interest can be simplified and may become more efficient. This may lead to a faster and more accurate identification of objects of interest.In some embodiments, no area of interest 16 is determined. In this case, the entire image is filtered for objects at or closer than the baseline depth.
[0114] The encroachment mask doesn’t distinguish between objects of interest, such as vegetation, and other objects within the area of interest. Therefore, objects of interest are now identified within the monocular image 10 based on the filtered depth map as shown by the highlighted region 18 in Figure 1. Specifically, using the depth information from the filtered depth map, objects which are encroaching on the railway may be identified. The encroaching objects are then checked whether they are of interest or not depending on the desired target (e.g., vegetation).
[0115] Identifying the objects of interest may comprise using further computer vision or machine learning techniques. For example, the process may involve using at least one of the following techniques: object detection; contour detection; colour filtering; local binary patterns; image classification; and semantic segmentation. In some embodiments, the identifying the objects of interest comprises using a second machine learning algorithm trained to identify the objects of interest.
[0116] Figures 4 and 5 show isolated objects 18, 19 within the monocular image 10 which have been determined to be encroaching on the railway based on the depth information from the filtered depth map (i.e., these objects are within the area of interest and within the baseline depth). In this embodiment, the target is to identify vegetation. Figure 4 shows the isolated object 18 which has been identified to be vegetation and therefore of interest. For this reason, this vegetation 18 has been highlighted in Figure 1 so that a user may readily see the objects of interest within the monocular image 10. Figure 5 shows the isolated object 19 which has been identified to be a fence and is therefore not of interest. For this reason, it has not been highlighted in Figure 1. It will be appreciated that, in some embodiments, the process involves identifying target objects other than vegetation in the monocular image 10.
[0117] In some embodiments, various metrics may be calculated from the detected objects of interest. The metrics may be displayed to a user and / or may be used to trigger warnings or actions based on the metrics.
[0118] In some embodiments, a coverage of the identified objects of interest within the monocular image may be calculated. An indication indicative of the coverage may be generated. Additionally or alternatively, a score may be assigned to the monocularimage 10 or a portion of the monocular image 10. The score indicates the level of encroachment on the railway and is based on the coverage. The score may be weighted based on a proximity of the identified objects of interest to the position of the rails or path of the train. In this way, the score may increase when the encroachment on the railway is greater.
[0119] Figure 6 shows an example embodiment in which the running rails 12 are used to divide the area of interest 16 into a plurality of distinct severity zones 17, each with their own distinct encroachment score and weighting.
[0120] For example, zone 17a comprises little identified vegetation and so has a coverage of 0.04%. Due to its low coverage and distance from the rails 12, its weighted score has been calculated to be 0. As a result, the coverage and the score are below any thresholds of interest and so this data may not be displayed to a user or no alert may be triggered on the basis of this zone. Zone 17b only contains an object which has been determined not to be vegetation, so is not of interest. Therefore, no metrics are calculated for this zone. Zone 17c comprises a significant portion of the identified vegetation of interest and has a coverage of 79.88%. However, as zone 17c is located at the edge of the area of interest 16 furthest from the rails 12, the severity of the encroachment of this vegetation is determined not to be high and so its weighted score is relatively low at 0.139. Zone 17d has a coverage of vegetation of 32.78%. However, due to its more central position within the area of interest 16 compared to zone 17c, zone 17d is closer to the path of the train. Thus, the weighted score is higher at 0.232. Zone 17e contains a low coverage of vegetation at 2.16% but it is weighted highly due to its position directly in the trains path in the centre of the monocular image 10. The weighted score of zone 17e is 0.109. It will be appreciated that the number and shape of the zones 17 may differ from the example shown in Figure 6.
[0121] In some example embodiments, the process is performed on a plurality of monocular images. In some embodiments, the results from multiple images may be condensed to make them more appropriate for human consumption. For example, a single coverage score may be generated which indicates a rolling average from a continuous sequence of monocular images.
[0122] In some embodiments, frames having a coverage or a weighted score over a threshold may be indicated to a user. In some embodiments, it is determined whether the coverage for each of the plurality of monocular images meets a threshold coverageand, if a number of monocular images which meets the threshold coverage meets a threshold number, an indication indicative of the presence of the objects of interest is generated. In other words, when the coverage is significant in multiple images, this may be a strong indicator that objects of interest are encroaching on the railway.
[0123] Therefore, when a certain number of images or a percentage of images from the plurality of images reaches a threshold, an indication of the potentially encroaching objects of interest is generated. In some embodiments, the indication may be generated when a rolling average coverage over successive image frames exceeds a configurable threshold.
[0124] Figure 7 illustrates steps in a method according to an embodiment.
[0125] In step S10, the method comprises obtaining a monocular image of a railway.
[0126] In step S20, the method comprises identifying a position of rails of the railway within the monocular image.
[0127] In step S30, the method comprises generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image.
[0128] In step S40, the method comprises selecting a reference point within the monocular image based on the position of the rails.
[0129] In step S50, the method comprises calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image.
[0130] In step S60, the method comprises filtering the depth map for objects at or closer than the baseline depth.
[0131] In step S70, the method comprises identifying objects of interest within the monocular image based on the filtered depth map.
[0132] Figure 8 shows an apparatus 100 comprising at least one processor 102 and at least one memory 104 storing instructions that, when executed by the at least one processor 102, cause the apparatus 100 to perform methods according to the embodiments above.A person of skill in the art would readily recognize that steps of various abovedescribed methods can be performed by programmed computers. Herein, some embodiments are also intended to cover program storage devices, e.g., digital data storage media, which are machine or computer readable and encode machineexecutable or computer-executable programs of instructions, wherein said instructions perform some or all of the steps of said above-described methods. The program storage devices may be, e.g., digital memories, magnetic storage media such as a magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media. The embodiments are also intended to cover computers programmed to perform said steps of the above-described methods. The tern non-transitory as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g. RAM vs ROM).
[0133] As used in this application, the term “circuitry” may refer to one or more or all of the following:
[0134] (a) hardware-only circuit implementations (such as implementations in analog, digital and / or quantum circuitry) and
[0135] (b) combinations of hardware circuit(s) and software, such as (as applicable):
[0136] (i) a combination of analog, digital and / or quantum hardware circuit(s) with software / firmware and
[0137] (ii) any or all portions of hardware processor(s) (including digital and / or quantum processor(s)) with software, and memory(ies) that work together to cause an apparatus, such as a mobile device, computing device or server, to perform various functions) and
[0138] (c) any or all portions of hardware circuit(s), such as microprocessor(s), processor(s), and / or quantum processor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.
[0139] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware.
[0140] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of twoor more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.
[0141] The ordering of method steps set out above may not be critical or fixed and the exact ordering of the steps may be varied as appropriate.
[0142] Although example embodiments of the present invention have been described in the preceding paragraphs with reference to various examples, it should be appreciated that modifications to the examples given can be made without departing from the scope of the invention as claimed.
[0143] Features described in the preceding description may be used in combinations other than the combinations explicitly described.
[0144] Although functions have been described with reference to certain features, those functions may be performable by other features whether described or not.
[0145] Although features have been described with reference to certain embodiments, those features may also be present in other embodiments whether described or not.
[0146] Whilst endeavouring in the foregoing specification to draw attention to those features of the invention believed to be of particular importance it should be understood that the Applicant claims protection in respect of any patentable feature or combination of features hereinbefore referred to and / or shown in the drawings whether or not particular emphasis has been placed thereon.
Claims
CLAIMS1. An apparatus comprising:at least one processor; andat least one memory storing instructions that when executed by the at least one processor cause the apparatus at least to perform:obtaining a monocular image of a railway;identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image;selecting a reference point within the monocular image based on the position of the rails;calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image;filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
2. An apparatus according to claim 1 , wherein the apparatus is further caused to perform:determining, based on the position of the rails, an area of interest within the monocular image in a plane of the baseline depth; andmasking the depth map using the area of interest such that information outside of the area of interest is discarded, wherein the filtering the depth map comprises filtering the masked depth map.
3. An apparatus according to claim 2, wherein the area of interest is a predetermined size and / or shape.
4. An apparatus according to claim 3, wherein the reference point is selected such that the area of interest fits within the monocular image at the baseline depth.
5. An apparatus according to any preceding claim, wherein the reference point is selected based on a quality of the image at the baseline depth.
6. An apparatus according to any preceding claim, wherein the reference point is selected based on calibration information relating to a camera used to obtain the monocular image.
7. An apparatus according to any preceding claim, wherein the identifying the position of the rails comprises using a first machine learning algorithm trained to identify rails in images.
8. An apparatus according to any preceding claim, wherein the monocular image is captured from a train travelling along the rails of the railway and the monocular image contains a plurality of sets of rails, the identifying the position of the rails comprises identifying the rails on which the train is travelling and ignoring any sets of rails on which the train is not travelling.
9. An apparatus according to any preceding claim, wherein the identifying the objects of interest comprises using at least one of the following:object detection;contour detection;colour filtering;local binary patterns;image classification; andsemantic segmentation.
10. An apparatus according to any preceding claim, wherein the identifying the objects of interest comprises using a second machine learning algorithm trained to identify the objects of interest.
11. An apparatus according to any preceding claim, wherein the objects of interest comprise vegetation.
12. An apparatus according to any preceding claim, wherein the apparatus is further caused to perform:calculating a coverage of the identified objects of interest within the monocular image; andgenerating a first indication indicative of the coverage.
13. An apparatus according to claim 12, wherein the first indication is weighted based on a proximity of the identified objects of interest to the position of the rails.
14. An apparatus according to any preceding claim, wherein the monocular image comprises a plurality of monocular images.
15. An apparatus according to claim 14, wherein the apparatus is further caused to perform:determining whether the coverage for each of the plurality of monocular images meets a threshold coverage; andif a number of monocular images which meets the threshold coverage meets a threshold number, generating a second indication indicative of the presence of the objects of interest.
16. A method comprising:obtaining a monocular image of a railway;identifying a position of rails of the railway within the monocular image; generating a depth map of the monocular image, the depth map indicating a distance to surfaces of objects within the monocular image;selecting a reference point within the monocular image based on the position of the rails;calculating a baseline depth based on a point in the depth map which corresponds to the reference point within the monocular image;filtering the depth map for objects at or closer than the baseline depth; and identifying objects of interest within the monocular image based on the filtered depth map.
