Method for generating training samples of foreign object defects of overhead power transmission line
By constructing a component segmentation network and a foreign object stable diffusion model, foreign object defect samples are generated using defect-free samples collected by UAVs. This solves the problem of insufficient foreign object defect samples in UAV inspection, improves the recognition accuracy, simulates foreign object deformation in real-world scenarios, and generates realistic training data.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HUAZHONG PHOTOELECTRIC TECH INST (CHINA SHIPBUILDING IND CORP THE NO 717 INST)
- Filing Date
- 2024-12-30
- Publication Date
- 2026-07-02
AI Technical Summary
Existing intelligent detection and recognition algorithms for defects in drone inspection images lack sufficient samples of foreign object defects, resulting in low recognition accuracy and difficulty in simulating the deformation of foreign objects in real-world scenarios.
By constructing a component segmentation network and a foreign object stable diffusion model, defect samples of foreign objects are generated using defect-free samples collected by UAVs. The PBD algorithm is then used to simulate the suspension and deformation of foreign objects, generating realistic images of foreign object defects, which are then labeled to expand the training dataset.
It generates a rich variety of foreign object defect samples, improves the recognition accuracy of the detection algorithm, and achieves a simulation effect close to that of real-world scenarios, solving the problems of small sample quantity and monotonous morphology.
Smart Images
Figure CN2024143640_02072026_PF_FP_ABST
Abstract
Description
Method for generating training samples for foreign object defects in overhead transmission lines
[0001] Related applications
[0002] This application claims priority to Chinese patent application No. 2024119518529, filed on December 27, 2024, entitled “Method for Generating Training Samples of Foreign Object Defects in Overhead Transmission Lines”. Technical Field
[0003] This invention relates to the field of deep learning technology, and more specifically to a method for generating training samples for foreign object defects in overhead power transmission lines. Background Technology
[0004] Foreign object defects in overhead transmission lines are one of the major hidden dangers threatening the safe and stable operation of the power grid. There are many kinds of foreign objects that may be suspended on the transmission lines, including broken plastic bags, woven bags, wires, rags, tree branches, bird nests, etc. These foreign objects may be suspended on overhead conductors and ground wires, poles and power components, and manual identification is costly in terms of manpower and resources.
[0005] With the development of neural network technology, it may be possible to identify these foreign objects through computer algorithms using neural network systems. However, the use of neural network models requires a large amount of data for training.
[0006] The image data captured in daily inspection tasks are mainly defect-free samples. The types of foreign object defects collected are limited, and the number is small and uneven. Therefore, there is a lack of necessary training data for developing intelligent defect detection and recognition algorithms for UAV inspections. Existing deep learning-based methods for detecting and recognizing defects in overhead transmission lines are prone to false detections and missed detections when dealing with foreign object defects.
[0007] Therefore, a sample augmentation technique for foreign object defects is needed to increase the number of foreign object defect samples, thereby making the foreign object defect samples in the dataset used to train the defect detection and recognition algorithm more diverse. Summary of the Invention
[0008] Purpose of the invention
[0009] The existing intelligent detection and recognition algorithms for defects in UAV inspection images require a large amount of sample data for training. However, the data collected in daily inspection tasks is mainly defect-free data, making it difficult to train defect detection algorithms with high generalization ability.
[0010] This invention proposes an effective solution that addresses the current problem of few and monotonous foreign object defect samples during UAV inspections. Furthermore, the generated foreign object model takes into account both the problem of foreign object suspension and the deformation of foreign objects under random wind forces, resulting in a richer variety of foreign object samples that are closer to the real situation.
[0011] This invention is based on a large number of defect-free samples collected during UAV inspections. It uses generative artificial intelligence to generate local images of foreign objects such as damaged woven bags, plastic bags, and rags of various shapes from the defect-free samples. The images are then fused into the original defect-free samples using an internal repair algorithm, thereby expanding the foreign object defect samples. At the same time, it generates annotation boxes for the generated foreign object defect areas to train the intelligent recognition algorithm.
[0012] To achieve the above objectives, the present invention adopts the following technical solution:
[0013] A method for generating simulated samples of foreign object defects in overhead transmission lines, the method comprising:
[0014] Step (1): Use a drone to fly along the overhead power transmission line and take pictures of the overhead power transmission line at a level angle to form a sample image of the defect-free overhead power transmission line.
[0015] Step (2): Construct a component segmentation network to extract masks of different key power components in the sample images. Each mask corresponds to a power component, and the coverage area of each power component forms a set of coordinate points.
[0016] Step (3): Traverse all pixels in the key power component mask extracted by the component segmentation network. For each pixel, determine the contact point between two power components by judging whether the 8-neighborhood of the pixel contains pixels of other power component masks. Determine whether each contact point belongs to the line suspension point according to the contact point category. Construct a set of line suspension point coordinates. Randomly select a line suspension point from the set of line suspension points for step (4).
[0017] Step (4): Use a planar mesh to perform a simple three-dimensional model of the foreign object. Randomly select a vertex of the planar mesh as the suspension point of the foreign object. Use the PBD algorithm to iteratively calculate the position and velocity of all vertices in the mesh under the combined action of gravity and random wind. After a predetermined number of iterations, fit the graphic area enclosed by the coordinates of all vertices on the planar mesh into a mask so that the suspension point of the foreign object in the mask coincides with the suspension point of the line.
[0018] Step (5) Generate a foreign object image based on the mask and foreign object information, fuse the foreign object image into the mask location of the sample image, and calculate the coordinate values of the bounding rectangle of the mask generated in step (3). Generate a foreign object annotation rectangle in the sample image and use the foreign object annotation rectangle to annotate the foreign objects in the sample image.
[0019] Furthermore, the neural network constructed in step (2) is the Mask R-CNN network.
[0020] Furthermore, it also includes the step of constructing a foreign matter stable diffusion model, which is based on the WIT, RedCaps, MMDialog, and CxC image-text public datasets.
[0021] Furthermore, the method also includes: randomly selecting several line suspension points from each line suspension point, and for each selected line suspension point, adding a foreign object diagram at the suspension position corresponding to the suspension point, so that the foreign object suspension point coincides with the line suspension point.
[0022] Furthermore, the power components include: wire clamps, ground wires, insulators, and connecting hardware.
[0023] Furthermore, the method also includes randomly adding images of different types of foreign objects to different sample images.
[0024] Furthermore, the process of generating the foreign object image in step (5) includes: inputting the generated mask into the foreign object stable diffusion model, randomly selecting foreign object type and pattern color description prompts from the prompt word library, and generating a foreign object image of the area corresponding to the mask based on the foreign object type and pattern color description prompts.
[0025] Furthermore, the method also includes selecting different suspension points and repeating steps (3)-(5) to generate different defect sample images. Beneficial effects
[0026] This invention can utilize a large number of defect-free UAV power line inspection samples to quickly generate a large number of realistic foreign object defect samples with diverse defect morphologies. This provides defect simulation training samples for the development of defect detection algorithms, thereby improving the detection accuracy of foreign object defects. Furthermore, this invention solves the problem of illogical generated defect images by using constructed possible suspension points as a benchmark to simulate defect samples that closely resemble real-world scenarios. Attached Figure Description
[0027] Figure 1 is a schematic flowchart of the method for generating simulated samples of foreign object defects in overhead transmission lines according to the present invention.
[0028] Figure 2 is a sample image of a defect-free drone power line inspection example;
[0029] Figure 3 is an example image of the power components in the sample image in Figure 2 after segmentation. The reference numerals for the segmentation in the figure represent: 1. Background; 2. Tower; 3. Insulator; 4. Equalizing ring; 5. Connecting hardware; 6-12. Conductor; 13-16. Wire clamp; 17-20. Vibration damper;
[0030] Figure 4 shows the set of suspension points selected from the power component mask (marked in white in the figure);
[0031] Figure 5 shows an example of the initial planar mesh used to simulate foreign objects, where P is a fixed suspension point of the foreign object;
[0032] Figure 6 shows the generated foreign object mask as an example, which is used to control the diffusion model to generate foreign objects (the diffusion model only superimposes random noise in the mask area, and then generates a high-quality local image by gradually reducing the noise in the mask area).
[0033] Figure 7 shows an example of a simulated foreign object defect. As can be seen from the figure, the simulated foreign object is very close to the actual suspension situation. Detailed Implementation
[0034] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0035] The process of generating training samples for foreign object defects in overhead transmission lines is shown in Figure 1.
[0036] As shown in the figure, the method for automatically generating training samples for foreign object defects in this embodiment includes the following process:
[0037] (1) Use a drone to fly along the overhead power transmission line and take pictures of the overhead power transmission line at eye level or any other required angle to form a sample image of the defect-free overhead power transmission line. When taking pictures, the sample images can be taken from both sides. Take pictures once every time the flight takes a period of time. There can be a certain overlap between the target image areas in the two shooting processes.
[0038] (2) Construct an instance segmentation network for the main components contained in the overhead transmission line. Using defect-free real inspection images, label key parts of the power poles, including clamps, conductors, ground wires, insulators, and connecting hardware, and establish a power component instance segmentation training set. Use the constructed power component instance segmentation training set to train the instance segmentation network model for power pole components.
[0039] For overhead transmission lines, the first step is to establish a database of key components and then label each key component according to the list in the database.
[0040] The instance segmentation network uses Mask R-CNN to segment the input inspection image and output a mask map of the power components, as shown in Figure 3. All collected defect-free inspection samples are input into the instance segmentation network for instance segmentation, extracting masks of key power components. The Mask R-CNN network is an improvement on the Faster R-CNN architecture, adding branches to predict the segmentation mask for each instance.
[0041] In one implementation, Mask R-CNN includes:
[0042] Backbone uses a pre-trained convolutional neural network (such as ResNet or ResNeXt) as a feature extractor to extract high-level feature maps from the input image.
[0043] The Region Proposal Network (RPN) is used to generate candidate regions of interest (RoIs), which are image regions that may contain objects. These RoIs are then used for more accurate object detection and segmentation.
[0044] RoI Align is used to more accurately extract the corresponding RoI features from the feature map through bilinear interpolation, avoiding information loss caused by quantization and thus improving the quality of mask prediction.
[0045] Classification and Bounding Box Regression Head: Used to classify each RoI (determine which category it belongs to) and adjust the position of the bounding box to better enclose the object;
[0046] Mask Prediction Head: For each RoI, in addition to classification and bounding box regression, there is a dedicated branch for predicting the binary mask corresponding to that RoI. This allows the model to provide pixel-level segmentation results for each detected object.
[0047] (3) Traverse all pixels in the key power component mask extracted by the instance segmentation network. For each pixel, determine the contact point between two power components by judging whether the 8-neighborhood of the pixel contains pixels of other power component masks. Determine whether each contact point belongs to the line suspension point according to the contact point category, construct the line suspension point coordinate set, and randomly select a line suspension point from the line suspension point set. Whether a contact point belongs to the suspension point is determined according to the contact point category, that is, according to the easy suspension position of the object in the line. For example, contact points between hardware, wires and clamps, and wires and vibration dampers are suspension points that are easy to suspend foreign objects and should be screened out.
[0048] By extracting the mask, the coordinates of various parts in the image that are prone to snagging foreign objects, such as the joints of various hardware, the contact points between the wire and the clamp, and the contact points between the wire and the vibration damper, are calculated.
[0049] For each sample generated, the coordinates of a line suspension point are randomly selected for subsequent matching with the coordinates of the foreign object suspension point.
[0050] (4) Select a suspension point from the possible suspension points of the foreign object, perform a simple three-dimensional model of the foreign object using a planar mesh, randomly select a vertex of the planar mesh as the suspension point of the foreign object, use the PBD algorithm to iteratively calculate the position and velocity of all vertices in the mesh under the combined action of gravity and random wind, and after a predetermined number of iterations, fit the graphic region enclosed by the coordinates of all vertices on the planar mesh into a mask.
[0051] For example, a simple 3D model of foreign objects such as damaged woven bags, plastic bags, and rags can be created using a planar mesh. A vertex in this mesh is randomly selected as a suspension point (as shown in Figure 5). The PBD algorithm (the specific process of which is described in patent US7616204) iteratively calculates the position and velocity of all vertices in the mesh under the combined effects of gravity and random wind. After a sufficient number of iterations, the graphic region enclosed by the coordinates of all vertices on the mesh is fitted into a mask. Morphological closing operations are used to smooth the mask, generating the final mask. The PBD algorithm can be executed using NVIDIA PhysX, Houdini, or Blender software. NVIDIA PhysX is a widely used physics engine that supports rigid body, soft body, cloth, and fluid simulations.
[0052] Specifically, the PBD algorithm updates the positions and velocities of all vertices in the planar mesh each time. As shown in Figure 5, in this embodiment, the mesh size is 10×10, the initial velocity of all vertices is 0, and after more than 50 iterations, the coordinates of all vertices no longer change significantly. The Alpha Shapes algorithm is used to recalculate the boundary of the point set composed of all vertices updated to the new coordinate positions, thus obtaining the mask. (The Alpha Shapes algorithm implementation is referenced from H. Edelsbrunner, D. Kirkpatrick and R. Seidel, "On the shape of a set of points in the plane," in IEEE Transactions on Information Theory, vol.29, no.4, pp.551-559, July 1983, doi:10.1109 / TIT.1983.1056714.)
[0053] The mask generated in step (3) is input into the foreign object stable diffusion model, and foreign object types (plastic bags, woven bags, rags, etc.) and pattern color description prompts are randomly selected from the prompt word library to generate a foreign object image of the specified area of the mask, and then superimposed and fused into the mask area of the original image (noise is superimposed on the mask area in the image to generate the target foreign object image, which is then fused with the background) to synthesize a foreign object defect sample (that is, the generated foreign object image is fused into the original sample image). The foreign object stable diffusion model is trained on image-text public datasets such as WIT, RedCaps, MMDialog, and CxC, as well as image-text pair datasets constructed from existing power grid inspection images. It is used to generate local foreign object images under the constraint of the mask area based on the input mask. That is, various image-text public datasets and foreign object-text pair datasets in power grid inspection images are obtained, the stable diffusion model is trained, and then a local foreign object image is generated using the stable diffusion model based on the foreign object type (plastic bags, woven bags, rags, etc.) and pattern color description prompts. For the construction of the stable diffusion model, see the article "Blended Diffusion for Text-driven Editing of Natural Images" by Omri Avrahami et al.
[0054] (5) Generate a bounding rectangle of the mask based on the mask and save it as a label file. Specifically, calculate the coordinate values of the bounding rectangle of the mask generated in step (3), randomly generate a foreign object labeling rectangle in the sample image, and add the foreign object labeling rectangle to the sample image so that the line suspension point and the foreign object suspension point coincide with each other.
[0055] Figure 7 shows the training samples obtained after adding foreign objects to a defect-free sample image.
[0056] Although the preferred embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many specific modifications under the guidance of the present invention without departing from the spirit of the invention and the scope of protection of the claims, and these modifications all fall within the scope of protection of the present invention.
Claims
1. A method for generating simulated samples of foreign object defects in overhead transmission lines, characterized in that, The method includes: Step (1): Use a drone to fly along the overhead power transmission line and take pictures of the overhead power transmission line at a level angle to form a sample image of the defect-free overhead power transmission line. Step (2): Construct a component segmentation network to extract masks of different key power components in the sample images. Each mask corresponds to a power component, and the coverage area of each power component forms a set of coordinate points. Step (3): Traverse all pixels in the key power component mask extracted by the component segmentation network. For each pixel, determine the contact point between two power components by judging whether the 8-neighborhood of the pixel contains pixels of other power component masks. Determine whether each contact point belongs to the line suspension point according to the contact point category. Construct a set of line suspension point coordinates. Randomly select a line suspension point from the set of line suspension points for step (4). Step (4): Use a planar mesh to perform a simple three-dimensional model of the foreign object. Randomly select a vertex of the planar mesh as the suspension point of the foreign object. Use the PBD algorithm to iteratively calculate the position and velocity of all vertices in the mesh under the combined action of gravity and random wind. After a predetermined number of iterations, fit the graphic area enclosed by the coordinates of all vertices on the planar mesh into a mask so that the suspension point of the foreign object in the mask coincides with the suspension point of the line. Step (5) Generate a foreign object image based on the mask and foreign object information, fuse the foreign object image into the mask location of the sample image, and calculate the coordinate values of the bounding rectangle of the mask generated in step (3). Generate a foreign object annotation rectangle in the sample image and use the foreign object annotation rectangle to annotate the foreign objects in the sample image.
2. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The neural network constructed in step (2) is the Mask R-CNN network.
3. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, It also includes the step of constructing a foreign matter stable diffusion model, which is constructed and trained based on the WIT, RedCaps, MMDialog, and CxC image-text public datasets.
4. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The method further includes: randomly selecting several line suspension points from each line suspension point, and for each selected line suspension point, adding a foreign object diagram at the suspension position corresponding to the suspension point, so that the foreign object suspension point coincides with the line suspension point.
5. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The electrical components include: wire clamps, ground wires, insulators, and connecting hardware.
6. The method for generating simulated foreign object defects in overhead transmission lines according to claim 5, characterized in that, The method also includes randomly adding images of different types of foreign objects to different sample images.
7. The method for generating simulated foreign object defects in overhead transmission lines according to claim 5, characterized in that, The process of generating the foreign object image in step (5) includes: inputting the generated mask into the foreign object stable diffusion model, randomly selecting foreign object type and pattern color description prompts from the prompt word library, and generating a foreign object image of the area corresponding to the mask based on the foreign object type and pattern color description prompts.
8. The method for generating simulated foreign object defects in overhead transmission lines according to claim 1, characterized in that, The method also includes selecting different suspension points and repeating steps (3)-(5) to generate different defect sample images.