Overhead wire fitting abnormality detection device and overhead wire fitting abnormality detection method

Abstract

This overhead wire fitting abnormality detection device comprises: at least one camera (9) that is installed on the roof of a vehicle (5) and captures images of an overhead wire fitting (7) ahead of and / or behind the vehicle (5) in the traveling direction thereof; a learning processing unit (13) that performs learning using image data captured by the camera (9); and a data processing unit (19) that performs abnormality determination using the image data. The learning processing unit (13) comprises: an equipment learning unit that learns detection of equipment; and a component learning unit that learns detection of a component of the equipment. The data processing unit (19) detects the equipment from the image data and determines that the image data is the best shot when the area in which the equipment has been detected is within a range of a predetermined size.

Description

Overhead Line Fitting Abnormality Detection Device and Overhead Line Fitting Abnormality Detection Method

[0001] The present invention relates to an overhead line fitting abnormality detection device and an overhead line fitting abnormality detection method.

[0002] Conventionally, there has been a maintenance inspection work for overhead line fittings in electric railway facilities, and an inspector visually confirmed abnormalities in the overhead line fittings and performed maintenance. An overhead line fitting refers to a fitting for holding an overhead trolley wire (trolley wire, hanging wire, auxiliary hanging wire; hereinafter referred to as "overhead line") for supplying power to a vehicle in electric railway facilities at a predetermined position, or for electrically connecting or dividing between a plurality of overhead lines. However, visual inspection has limitations and there is a possibility of overlooking abnormalities, so there has been a demand for automation of this inspection. Furthermore, there is also a demand to perform the inspection work of overhead line fittings regardless of day or night during the actual running of operating vehicles on the live line.

[0003] Therefore, in Patent Document 1, an object is photographed with a camera mounted on a moving body, an image is generated using a generation model such as a neural network using the photographed image, the similarity between the generated image and the photographed image is obtained, and when this similarity is small, it is determined that there is an abnormality.

[0004] In Patent Document 2, in order to improve the accuracy of abnormality detection, learning is performed for each part constituting an overhead line fitting, and an overhead line fitting abnormality detection device that performs abnormality detection is disclosed. In this device, first, an overhead image of a railway vehicle is acquired by a camera mounted on the roof of the railway vehicle. Next, the overhead line fittings learned by object detection of machine learning are detected from the acquired image, and semantic segmentation is applied to the target equipment image to convert it into a one-hot representation. This simplifies the image and simplifies the abnormality determination of shape abnormalities. Since it is possible to extract only the regions of each part of the fitting by semantic segmentation, a mask image of only the part is generated using this, normal data learning for each part is performed, and an abnormality determination is made using the restoration error of the mask part.

[0005] Japanese Patent Application Laid-Open No. 2019-133306 Japanese Patent Application Laid-Open No. 2022-159010

[0006] The technologies disclosed in Patent Documents 1 and 2 perform anomaly detection processing on all images captured by the camera. When these technologies are applied to forward monitoring, the overhead line equipment is photographed in the direction of the rails (forward, backward, or both). In this case, the same equipment is photographed in consecutive frames. Therefore, if anomaly detection is performed on all frames, the same equipment will undergo anomaly detection processing multiple times, which may result in wasted processing. Furthermore, when anomaly detection is performed on images in which the target object is small, the number of pixels in the area of ​​the target object decreases, leading to a decrease in detection accuracy. Conversely, when anomaly detection is performed on images in which the target object is too large, the target equipment may be cut off, resulting in a decrease in detection accuracy similar to when the target object is small.

[0007] This invention has been made in view of the circumstances described above, and the problem that this invention aims to solve is to provide an overhead line fitting abnormality detection device and overhead line fitting abnormality detection method that can reduce the load of abnormality detection processing and improve abnormality detection accuracy by selecting and processing the best image from among images of the same equipment detected by equipment detection.

[0008] The overhead line fitting abnormality detection device according to the present invention, which solves the above problems, comprises at least one camera installed on the roof of a vehicle that photographs overhead line fittings in front of and / or behind the vehicle in the direction of travel, a learning processing unit that learns using image data captured by the cameras, and a data processing unit that performs abnormality determination using the image data. The learning processing unit comprises an equipment learning unit that learns to detect equipment and a parts learning unit that learns to detect parts of the equipment. The data processing unit detects the equipment from the image data using an equipment detection model learned by the equipment learning unit, determines the image data as a best shot if the area in the image data where the equipment is detected is within a predetermined size range, detects the parts from the equipment in the image data determined to be a best shot using a parts detection model learned by the parts learning unit, and determines abnormalities in the detected equipment and / or parts using a pre-learned abnormality detection model.

[0009] The overhead line fitting abnormality detection method of the present invention includes: photographing overhead line fittings in front of and / or behind the direction of travel of the vehicle with at least one camera installed on the roof of the vehicle; performing learning using image data captured by the camera; and performing abnormality determination using the image data, wherein the learning includes learning to detect equipment and learning to detect parts of the equipment, and the abnormality determination includes detecting the equipment from the image data using an equipment detection model learned by learning to detect the equipment, determining the image data as a best shot if the area in the image data in which the equipment is detected is within a predetermined size range, detecting the parts from the equipment in the image data determined as a best shot using a part detection model learned by learning to detect the parts, and determining abnormalities in the detected equipment and / or parts using a pre-learned abnormality detection model.

[0010] According to this invention, by selecting and processing the best image from among images of the same equipment detected by equipment detection, it is possible to provide an overhead line fitting abnormality detection device and an overhead line fitting abnormality detection method that can reduce the load of abnormality detection processing and improve the accuracy of abnormality detection.

[0011] This is a schematic diagram illustrating the implementation status of the overhead wire fitting abnormality detection device according to an embodiment of the present invention. This is a block diagram illustrating the configuration learning processing unit of the overhead wire fitting abnormality detection device according to an embodiment of the present invention. This is a schematic diagram showing an example of a camera image according to an embodiment of the present invention. This is a flowchart illustrating the operation of the overhead wire fitting abnormality detection device according to an embodiment of the present invention. This is a flowchart illustrating the operation of equipment detection learning according to an embodiment of the present invention. This is a schematic diagram illustrating equipment detection results according to an embodiment of the present invention. This is a flowchart illustrating the operation of component detection learning according to an embodiment of the present invention. This is a schematic diagram illustrating component detection results according to an embodiment of the present invention. This is a schematic diagram illustrating image cropping and resizing results according to an embodiment of the present invention. This is a flowchart illustrating the operation of abnormality detection learning according to an embodiment of the present invention. This is a schematic diagram illustrating abnormality detection due to restoration error according to an embodiment of the present invention.

[0012] The embodiments of the present invention will now be described with reference to the attached drawings. Figure 1 is a schematic diagram showing the implementation of the overhead wire fitting abnormality detection device 1 according to an embodiment of the present invention. As shown in Figure 1, the overhead wire fitting abnormality detection device 1 according to the present invention is mounted on a vehicle 5 traveling on rails 3, and cameras 9 (9a, 9b) installed on the roof of the vehicle 5 photograph the overhead wire fittings 7 in front of and behind the direction of travel of the vehicle, and detect abnormalities in the overhead wire fittings 7 based on the obtained image data. In this embodiment, two cameras are used for front and rear photography, but the invention is not limited to this, and one or more cameras, or two or more cameras, may be installed with different shooting angles and directions.

[0013] Figure 2 is a block diagram showing the configuration of the overhead wire fitting abnormality detection device 1. Figure 3 is a block diagram showing the configuration of the learning processing unit 13 of the overhead wire fitting abnormality detection device 1. As shown in Figures 2 and 3, the overhead wire fitting abnormality detection device 1 according to this embodiment includes a camera 9, a storage device 11, a learning processing unit 13, a data processing unit 19, and an input / output unit 21 (user interface, UI). The learning processing unit 13 includes an equipment learning unit 15 and a component learning unit 17.

[0014] As mentioned above, camera 9 is installed on the roof of vehicle 5 and photographs the overhead line fittings 7 in front of and / or behind the vehicle 5 in the direction of travel. In this embodiment, camera 9a photographs the area in front of the vehicle in the direction of travel and camera 9b photographs the area behind the vehicle in the direction of travel. In this embodiment, camera 9 is an area sensor camera and uses visible light illumination or infrared light illumination as needed, but what is required is a means to photograph the overhead line equipment as shown in the example surveillance image in Figure 4, which will be described later, and the configuration is not limited. The storage device 11 stores the image data captured by cameras 9a and 9b. Here, the storage device 11 may be implemented as a storage device such as a semiconductor memory or a magnetic disk.

[0015] The learning processing unit 13 generates a learning model using deep learning. Here, the equipment learning unit 15 learns equipment detection using image data stored in the storage device 11 and generates an equipment detection model. The parts learning unit 17 learns equipment parts detection using the image data from which the equipment was detected and generates a parts detection model.

[0016] The data processing unit 19 detects the equipment from the aforementioned image data using the equipment detection model learned by the equipment learning unit 15. At this time, if the area in the image data where the equipment is detected is within a predetermined size range, this image data is determined to be the best shot. Next, from the equipment in the image data determined to be the best shot, the parts are detected using the parts detection model learned by the parts learning unit 17, and abnormalities in the detected equipment and / or parts are determined using a pre-learned abnormality detection model.

[0017] In this embodiment, the learning processing unit 13 and the data processing unit 19 may be information processing devices such as a personal computer or a server configured on the cloud. Each function, including the equipment learning unit 15 and the component learning unit 17, may be software or programs executed by a processor such as a CPU, GPU (Graphics Processing Unit), and TPU (Tensor Processing Unit) within the information processing device. The input / output unit 21 includes an input unit including at least one of a pointing device and a keyboard, and an output unit including a display.

[0018] Figure 5 is a flowchart illustrating the operation of the overhead line fitting abnormality detection device 1 according to this embodiment. The operation of the overhead line fitting abnormality detection device 1 will be explained using this flowchart.

[0019] <Image Input> The overhead line fittings 7 are photographed by a camera 9 mounted on the roof of the vehicle 5 (Figure 5, S001). <Storage in Storage Device> The image data captured by the camera 9 is stored in the storage device 11 (Figure 5, S002).

[0020] <Equipment Detection> Target equipment is detected using the equipment detection model from the equipment detection model database M01 (Figure 3, S003). The equipment detection model has been pre-trained with deep learning. Figure 6 is a flowchart showing the learning operation of the equipment detection model executed by the equipment learning unit 15. Figure 7 is a schematic diagram showing the result of detecting target equipment using the equipment detection model MS. As shown in Figure 6, the equipment learning unit 15 first reads image data from the driving image dataset D13 (Figure 6, S101). Next, equipment detection learning is performed using data from the equipment annotation database D14, which includes two pieces of data: position information of the rectangular region surrounding the target equipment in the image data (coordinates of the four vertices of the rectangular region) and the correct label (Figure 6, S102). The deeply trained equipment detection model is stored in the equipment detection model database M01 (Figure 6, S103).

[0021] Figure 7 illustrates the operation of equipment detection using an equipment detection model MS that has been pre-trained by deep learning. The equipment detection model MS receives an input image G11 and outputs an equipment detection result G12. As part of the equipment detection result, the "rectangular coordinate information of the equipment" is obtained as "rectangular coordinate information in the image data" indicated by the dashed line A in the equipment detection result G12, and the "equipment label" corresponding to the type of equipment is obtained as "label information".

[0022] <Equipment Detection Result Determination> Next, if equipment is detected, proceed to the best shot determination (Figure 5, S004, YES). If no equipment is detected, proceed to the shooting completion determination (Figure 5, S004, NO).

[0023] <Best Shot Determination> Based on the label information of the detected equipment, the best shot rectangle size threshold for that equipment label is read from the best shot database D01 and compared. If the rectangle size (image area) of the equipment falls within the threshold area, it is determined to be the best shot and the process proceeds to component determination (Figure 5, S005, YES). If it is determined that it is not the best shot, the process proceeds to the next image (Figure 5, S005, NO).

[0024] Let's explain the best shot determination further here. Figure 4 is a schematic diagram showing an example of a camera image (surveillance image). The frames F1 to F3 shown on the upper left of the figure are the results of the rear camera 9b in the direction of travel of the vehicle 5, and equipment detection is performed as shown in rectangles A1 to A3. The frames F4 to F6 shown on the upper right of the figure are the results of the front camera 9a in the direction of travel of the vehicle 5, and equipment detection is performed as shown in rectangles A4 to A6. Performing the subsequent processing (a series of anomaly detection processes) following equipment detection for all frames in which equipment is detected would result in the same processing being performed multiple times for the same equipment, which would increase computational costs and lead to duplicate detection results, and is not desirable from the standpoint of processing efficiency. Also, if the target equipment is detected in a small rectangle, the number of pixels of the target equipment will be small, and the detection accuracy tends to decrease in proportion to the resolution. Furthermore, if the target equipment is detected in a rectangle that is too large, there is a possibility that the target equipment is cut off from the image, and similarly, the detection accuracy may decrease.

[0025] Based on the above considerations, in order to improve processing efficiency and detection accuracy, images obtained from prior trial experiments will be analyzed, and appropriate maximum and minimum threshold values ​​will be set for the areas where equipment is detected. If the detected equipment area falls within the predetermined threshold range, the image will be judged as the best shot. Specifically, maximum and minimum threshold values ​​will be set for both the vertical and horizontal dimensions of the rectangle, and if the detected rectangle falls within these thresholds, it will be judged as the best shot. Alternatively, the maximum and minimum values ​​of the number of pixels in the detected equipment area may be set as the threshold values.

[0026] <Part Existence Determination> Based on the "equipment label" and inspection item database D02 when equipment is detected in equipment detection (Figure 5, S003), it is determined whether the target part exists in the detected equipment (Figure 5, S006). The inspection item database D02 contains information on whether the target part exists in each equipment label. If the target part exists, the process proceeds to part detection (Figure 5, S006, YES); otherwise, the process proceeds to image cropping and resizing (Figure 5, S006, NO).

[0027] <Part Detection> In part detection, object detection is performed only within the rectangular area detected by equipment detection (Figure 5, S003). At this time, part detection is performed using a part detection model from the part detection model database M02 for each target piece of equipment in which the target part exists (Figure 5, S007). The part detection model has been pre-trained with deep learning. Figure 8 is a flowchart showing the learning operation of the part detection model executed by the part learning unit 17. Figure 9 is a schematic diagram showing the result of detecting a target part using the part detection model MB, which is illustrated as a concept. As shown in Figure 8, the part learning unit 17 first reads image data from the equipment image dataset D15 (Figure 8, S201). Next, it performs part detection learning using data from the part annotation database D16, which includes two pieces of data: position information of the rectangular area surrounding the target part in the image data (coordinates of the four vertices of the rectangular area) and the correct label (Figure 8, S202). The deep-trained part detection model is stored in the part detection model database M02 (Figure 8, S203). Next, if the learning of all equipment components is not yet complete (Figure 8, S204, NO), the same flow is repeated again from image input S201. If the learning of all equipment components is complete (Figure 8, S204, YES), the learning operation ends there.

[0028] Figure 9 illustrates the operation of equipment detection using a pre-trained deep learning-based component detection model MB. The component detection model MB receives the input image G13 and outputs the component detection result G14. The component detection result obtains multiple "rectangular coordinate information of components" indicated by the dashed line B in the component detection result G14 as "rectangular coordinate information in image data," and a "component label" corresponding to the type of component as "label information." Here, the detection result retains the detection results of both equipment and components, such as the "rectangular coordinate information of equipment" and "equipment label" inherited from equipment detection, and the "rectangular coordinate information of components" and "component label" detected in component detection. Note that the figure illustrated here differs in shape from the equipment illustrated earlier in this embodiment, but the operation of component detection is equivalent. Similarly, the figures shown for illustrative purposes here will have different shapes, but the operation will be equivalent.

[0029] <Image Extraction and Resizing> From the image data, an image is extracted based on the "rectangular coordinate information of the equipment" shown by the dashed line A in Figure 7, or the "rectangular coordinate information of the parts" shown by the dashed line B in Figure 9 (Figure 5, S008), and then resized to an image size that can be handled by the anomaly detection model (Figure 5, S009). Figure 10 is a schematic diagram showing an example of the processing result of image extraction and resizing. In Figure 10, image extraction and resizing of the rectangular coordinate area of ​​dashed line A are exemplified for the equipment detection result. The extracted image is resized to a square for object detection. The "rectangular coordinate information in the image data," "label information," and "resized extracted image" are sent to the subsequent processing.

[0030] <Image Correction Processing> In the image correction processing, the image data that was extracted in the previous stage and resized to an image size that can be handled by the anomaly detection model is corrected (Figure 5, S010). Image correction processing may include, for example, brightness correction processing and edge enhancement processing. At this time, the method of brightness correction and edge enhancement processing can be any method, but examples include gamma correction and contrast correction for brightness correction, and bilateral filters for edge enhancement. The corrected "corrected image" is sent to the next image inversion process along with "rectangular coordinate information" and "label information".

[0031] <Image Inversion Processing> In the image inversion processing, the left-right inversion target area information is read from the left-right inversion target area information database (Figure 5, D03), and it is checked whether the position of the detected object in the corrected image data received from the previous image correction processing (Figure 5, S010) belongs to the left-right inversion target area (Figure 5, S011). If the position of the detected object belongs to the left-right inversion target area (Figure 5, S011, YES), the image inversion processing is performed (Figure 5, S012), and the inverted image data is sent to the anomaly degree calculation processing (Figure 5, S013). If the detection position of the detected object does not belong to the left-right inversion target area (Figure 5, S011, NO), the image-corrected image data is sent as is to the anomaly degree calculation processing (Figure 5, S013).

[0032] Here, we will briefly explain image inversion processing. The equipment and parts targeted for anomaly detection may have symmetry. If we call the parts that have symmetry the "right part" and the "left part," then without inversion processing, it would be necessary to train the anomaly detection model separately for the "right part" and the "left part." However, since the "right part" inherently has a structure equivalent to the "left part" when inverted, we can distinguish between the "right part" and the "left part" based on the position of the detected part, and by inverting one of them, we can arrange the image set so that anomaly detection can be performed with the same structure. For example, an anomaly detection model can be generated by training only the "right part" or only the "left part," thereby reducing the computational cost of model generation. This embodiment employs such image inversion processing.

[0033] <Calculation of Anomaly Score> The anomaly detection model for each piece of equipment or part, which has been previously trained, is read from the anomaly detection model database M03, applied to the resized cropped image, and the anomaly score is calculated (Figure 5, S013). The anomaly detection model has been pre-trained using deep learning. Figure 11 is a flowchart showing the training operation of the anomaly detection model performed by the equipment training unit 15 and the part training unit 17. Figure 12 is a schematic diagram showing the result of calculating the anomaly score using the anomaly detection model MK. As shown in Figure 11, the equipment training unit 15 and the part training unit 17 first read image data from their respective anomaly detection image datasets D17 (Figure 11, S301). Anomaly detection training is performed on the read images (Figure 11, S302). The deeply trained anomaly detection model is saved in the anomaly detection model database M03 (Figure 11, S303). Next, if anomaly detection training for all equipment and parts has not been completed (Figure 11, S304, NO), the same flow is repeated again from image input S301. Once the learning process for detecting anomalies in all equipment and components is complete (Figure 11, S304, YES), the learning process ends.

[0034] Figure 12 illustrates the operation of anomaly detection using an anomaly detection model MK, which has been pre-trained by deep learning. The anomaly detection model MK takes input images G15 and G16 as input and outputs the component detection result G17. Here, G15 is an example of normal data and G16 is an example of abnormal data. Here, a distance-based anomaly detection model using multidimensional features is illustrated as an example of the anomaly detection model. The anomaly detection model used here can use either supervised or unsupervised training data, as long as it outputs multidimensional features, but if supervised training data is used, it is necessary to annotate whether it is normal or abnormal beforehand. The output multidimensional features are plotted in the latent variable space, and the anomaly score is calculated based on the distance from the normal center. Abnormal data is plotted at a position relatively far from the normal center.

[0035] <Anomaly Determination> The anomaly degree, calculated as a distance in the latent variable space, is read from the previously created anomaly detection threshold database D04, and an anomaly determination is performed using the anomaly determination threshold set for each target equipment and target part (Figure 5, S014). As mentioned above, if the output is a multidimensional feature, the anomaly degree is calculated as the distance from the normal center, and if the anomaly degree is greater than or equal to the threshold (Figure 5, S014, YES), an anomaly flag is assigned (Figure 5, S016). If the anomaly degree is not greater than or equal to the threshold (Figure 5, S014, NO), a normal flag is assigned (Figure 5, S015). After each flag is assigned, the anomaly detection result is saved (Figure 5, S017). The information saved here is "rectangular coordinate information", "label information", and "anomaly determination result". <Acquisition Completion Determination> Finally, it is determined whether processing has been completed for all images (Figure 5, S018). If acquisition has not been completed (Figure 5, S018, NO), the series of flows from image input S001 is repeated. If the shooting is complete (Figure 5, S018, YES), the process will be terminated.

[0036] Furthermore, while there are no particular limitations on the deep learning methods used in the detection models for equipment detection (Figure 3, S003) and component detection (Figure 3, S007) in this processing flow, examples include SSD (SingleShot Multibox Detector).

[0037] Up to this point, we have used a distance space threshold based on multidimensional features as the anomaly detection model, but we are not limited to this, and we may also use a reconstruction error calculated from the input image and the image reconstructed by an autoencoder or the like. Figure 13 is a schematic diagram illustrating anomaly detection using reconstruction error. For example, when using reconstruction error, the anomaly detection model MA consists of an encoder and a decoder, and takes images G18 and G19 as input and outputs reconstructed images G20 and G21. Then it compares the difference between the input images G18 and G19 and the reconstructed images G20 and G21 and outputs the result. For difference comparison, for example, the difference in RGB values ​​for each pixel may be calculated and these may be summed up and used as an index of the degree of anomaly.

[0038] As described above, in this embodiment, equipment detection is performed using an equipment detection model, and then component detection is performed on the detected equipment using a component detection model for each piece of equipment. Since the degree of abnormality is calculated for each detected piece of equipment and component using its respective abnormality detection model, abnormality detection can be performed not only on equipment with a relatively large area but also on components with a small area, thereby improving the accuracy of abnormality detection. By performing object detection in stages, from equipment detection to component detection, it is possible to detect relatively small areas without using semantic segmentation, and by employing object detection with low computational and annotation costs for target component detection, it is possible to reduce computational costs and improve detection accuracy. Furthermore, after equipment detection, the best shot is determined based on preset maximum and minimum threshold values ​​for the detected equipment area, and the subsequent abnormality detection processing is performed on the image determined to be the best shot. Therefore, it is possible to provide an overhead line fitting abnormality detection device and overhead line fitting abnormality detection method that can reduce the load of abnormality detection processing and improve abnormality detection accuracy.

[0039] In addition to the above, it is possible to select or replace the configurations listed in the above embodiments, or to change them to other configurations as appropriate, as long as it does not deviate from the spirit of the present invention.

Claims

1. An overhead line fitting abnormality detection device comprising: at least one camera mounted on the roof of a vehicle and photographing overhead line fittings in front of and / or behind the vehicle in the direction of travel; a learning processing unit that performs learning using image data captured by the cameras; and a data processing unit that performs abnormality determination using the image data, wherein the learning processing unit comprises an equipment learning unit that learns to detect equipment and a parts learning unit that learns to detect parts of the equipment; the data processing unit detects the equipment from the image data using an equipment detection model learned by the equipment learning unit; determines the image data as a best shot if the area in the image data in which the equipment is detected is within a predetermined size range; detects the parts from the equipment in the image data determined to be a best shot using a parts detection model learned by the parts learning unit; and determines an abnormality in the detected equipment and / or parts using a pre-learned abnormality detection model.

2. An overhead line fitting abnormality detection method comprising: photographing overhead line fittings in front of and / or behind the direction of travel of the vehicle with at least one camera installed on the roof of the vehicle; performing learning using image data captured by the camera; and performing abnormality determination using the image data, wherein the learning includes learning to detect equipment and learning to detect parts of the equipment; and the abnormality determination includes detecting the equipment from the image data using an equipment detection model learned by learning to detect the equipment; determining the image data as a best shot if the area in the image data in which the equipment is detected is within a predetermined size range; detecting the parts from the equipment in the image data determined to be a best shot using a part detection model learned by learning to detect parts; and determining an abnormality of the detected equipment and / or parts using a pre-learned abnormality detection model.

Images