Road damage detection apparatus and method, and electronic apparatus

Abstract

Provided in the embodiments of the present application are a road damage detection apparatus and method, and an electronic device. The road damage detection apparatus comprises: an extraction apparatus, which extracts from an image a first region including a road, wherein the area of the first region is smaller than the area of the image; a detection apparatus, which respectively inputs the first region into a plurality of detection models in the detection apparatus, wherein each detection model outputs a detection result for the first region; and a post-processing apparatus, which performs processing on the detection results respectively outputted by the plurality of detection models, so as to generate a detection result reflecting damages on the road. In the road damage detection apparatus of the present application, a plurality of detection models are used to perform road damage detection on an extracted road region image, and the plurality of detection models can detect road damages of different shapes and / or different sizes, thereby improving the accuracy of road damage detection.

Description

Road damage detection devices and methods and electronic equipment Technical Field

[0001] This application relates to the field of image processing technology, and more particularly to a road damage detection device and method, as well as an electronic device. Background Technology

[0002] Road damage poses various safety hazards, such as vehicle steering or loss of control, which can lead to safety accidents.

[0003] Current road damage detection technologies mainly focus on vehicle-mounted cameras. These technologies can be broadly divided into two categories: the first is rule-based image processing technology, which lacks universality for different environments (e.g., different backgrounds or different lighting); the second is data-driven road damage detection technology, which uses data-intensive deep learning models, which are either fully convolutional, attention-based, or a combination of both, requiring the training of millions of parameters.

[0004] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and for the convenience of those skilled in the art to understand them. It should not be assumed that the above technical solutions are known to those skilled in the art simply because these solutions have been described in the background section of this application.

[0005] Summary of the Invention

[0006] Surveillance cameras installed near roads are typically used to monitor traffic conditions and identify potential hazards, and they have been widely deployed in various locations. However, surveillance cameras are usually installed at high angles, making it difficult for existing road damage detection technologies to detect road damage such as small cracks and potholes from the perspective of the surveillance cameras.

[0007] To address at least one of the aforementioned technical problems, embodiments of this application provide a road damage detection device, method, and electronic device. In this road damage detection method, multiple detection models are used to detect road damage in extracted road area images. The multiple detection models can detect road damage of different shapes and / or different sizes, thereby improving the accuracy of road damage detection. For example, accurate road damage detection can also be performed based on images acquired by surveillance cameras.

[0008] According to one aspect of the embodiments of this application, a road damage detection device is provided, the device comprising:

[0009] An extraction device extracts a first region containing a road from an image, the area of ​​which is smaller than the area of ​​the image.

[0010] The detection device inputs the first region into multiple detection models, wherein each detection model outputs a detection result for the first region; and

[0011] The post-processing device processes the detection results output by each of the multiple detection models to generate detection results reflecting the damage on the road.

[0012] According to another aspect of the embodiments of this application, a road damage detection method is provided, the method comprising:

[0013] Extract a first region containing roads from the image, the area of ​​the first region being smaller than the area of ​​the image;

[0014] The first region is input into multiple detection models, and each detection model outputs a detection result for the first region; and

[0015] The detection results output by each of the multiple detection models are processed to generate detection results reflecting the damage on the road.

[0016] According to another aspect of the present application, an electronic device is provided, including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to implement the road damage detection method as described above.

[0017] One of the beneficial effects of the embodiments of this application is that: in the road damage detection method of this application, multiple detection models are used to detect road damage in the extracted road area image. The multiple detection models can detect road damage of different shapes and / or different sizes, which can improve the accuracy of road damage detection.

[0018] Referring to the following description and accompanying drawings, specific implementation methods of the embodiments of this application are disclosed in detail, indicating how the principles of the embodiments of this application can be adopted. It should be understood that the implementation methods of this application are not limited in scope. Within the spirit and scope of the appended claims, the implementation methods of this application include many changes, modifications, and equivalents. Attached Figure Description

[0019] The accompanying drawings, which form part of the specification, are used to provide a further understanding of the embodiments of this application and illustrate the implementation methods of this application, together with the textual description, to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other implementation methods based on these drawings without creative effort. In the drawings:

[0020] Figure 1 is a schematic diagram of a road damage detection method according to an embodiment of this application;

[0021] Figure 2 is another schematic diagram of the road damage detection method according to an embodiment of this application;

[0022] Figure 3 is a schematic diagram of a first region extraction method according to an embodiment of this application;

[0023] Figure 4 is a schematic diagram of an image captured by a surveillance camera according to an embodiment of this application;

[0024] Figure 5 is a schematic diagram of a semantic segmentation mask image according to an embodiment of this application;

[0025] Figure 6 is a schematic diagram of the detection of the first region in an image captured by a surveillance camera according to an embodiment of this application;

[0026] Figure 7 is a schematic diagram of the first region extracted according to an embodiment of this application;

[0027] Figure 8 is a schematic diagram of a post-processing method according to an embodiment of this application;

[0028] Figure 9 is a schematic diagram of a road damage detection process according to an embodiment of this application;

[0029] Figure 10 is a schematic diagram of a method for synthesizing road damage samples according to an embodiment of this application;

[0030] Figure 11 is a schematic diagram of road damage images;

[0031] Figure 12 is a schematic diagram of the road damage graphic annotation in Figure 11;

[0032] Figure 13 is a schematic diagram of a road damage sample synthesized based on the road damage graphic annotations in Figure 12;

[0033] Figure 14 is another schematic diagram of road damage images;

[0034] Figure 15 is a schematic diagram of the road damage graphic annotation in Figure 14;

[0035] Figure 16 is a schematic diagram of a road damage sample synthesized based on the road damage graphic annotations in Figure 15;

[0036] Figure 17 is a schematic diagram of a real-life image captured by a surveillance camera;

[0037] Figure 18 is a schematic diagram of a road damage sample synthesized using the method of the embodiments of this application;

[0038] Figure 19 is a schematic diagram of a road damage detection device according to an embodiment of this application;

[0039] Figure 20 is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0040] Referring to the accompanying drawings, the foregoing and other features of the embodiments of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial implementations in which the principles of the embodiments of this application can be adopted. It should be understood that this application is not limited to the described embodiments; rather, the embodiments of this application include all modifications, variations, and equivalents falling within the scope of the appended claims.

[0041] In the embodiments of this application, the terms "first," "second," etc., are used to distinguish different elements by name, but do not indicate the spatial arrangement or chronological order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in association and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0042] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0043] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments. The term "comprising / including" as used herein means the presence of a feature, integral, step, or component, but does not exclude the presence or addition of one or more other features, integrals, steps, or components.

[0044] First aspect of the embodiments

[0045] This application provides a method for detecting road damage.

[0046] Figure 1 is a schematic diagram of a road damage detection method according to an embodiment of this application. As shown in Figure 1, the road damage detection method includes:

[0047] Operation 101: Extract a first region containing roads from the image, the area of ​​the first region being smaller than the area of ​​the image;

[0048] Operation 102: Input the first region into multiple detection models, wherein each detection model outputs a detection result for the first region; and

[0049] Operation 103: Process the detection results output by each of the multiple detection models to generate detection results reflecting the damage on the road.

[0050] According to an embodiment of the first aspect of this application, in road damage detection, multiple detection models are used to detect road damage in extracted road area images. These multiple detection models can detect road damage of different shapes and / or sizes, thereby improving the accuracy of road damage detection. The method of this application is particularly effective in improving the detection results for road damage such as small cracks and potholes.

[0051] In operation 101 of this application, the image can be an image from a traffic monitoring video, such as an image from a video or image sequence captured by a monitoring camera installed near the road for monitoring traffic conditions.

[0052] Surveillance cameras are typically installed at high positions, capturing images of the road from high angles or long distances. When using existing technologies to detect road damage from such images, it is difficult to accurately detect small cracks or potholes, resulting in generally poor detection performance. The road damage detection method of this application improves the accuracy of road damage detection, enabling accurate detection even from images acquired by surveillance cameras. Furthermore, compared to using images captured by vehicle-mounted cameras for road damage detection, this application integrates the proposed method into existing traffic monitoring and analysis systems, reducing the cost of road damage detection by utilizing images acquired by surveillance cameras.

[0053] In the following description of this application, the example will be taken as an image from a traffic surveillance video, but this description also applies to cases where the image is from other sources. Furthermore, this method is not limited to scenarios with high shooting angles or long shooting distances, but is also applicable to scenarios with other shooting angles or distances.

[0054] Figure 2 is another schematic diagram of the road damage detection method according to an embodiment of this application. As shown in Figure 2, the road damage detection method includes:

[0055] Operation 101: Extract a first region containing roads from the image, the area of ​​the first region being smaller than the area of ​​the image;

[0056] Operation 201: Perform two or more preprocessing steps on the first region to generate multiple preprocessed images;

[0057] Operation 202: Input the multiple preprocessed images into each of the multiple detection models respectively, wherein each detection model outputs the detection result for each preprocessed image;

[0058] Operation 103: Process the detection results output by each of the multiple detection models to generate detection results reflecting the damage on the road.

[0059] The descriptions of operations 101 and 103 in Figure 2 are the same as those of operations 101 and 103 in Figure 1.

[0060] In operation 101 of this application, a first region is extracted from the image, which includes a road. The first region is smaller than the area of ​​the image, so that when detecting road damage, the detection can be targeted at the first region, avoiding interference from similar potholes, wrinkles, and other patterns in other regions, thus improving the accuracy of road damage detection. Furthermore, reducing the area to be detected also improves detection efficiency.

[0061] In operation 101, a first region in the image can be extracted based on the determination result of whether the image is the first frame in a set of images. For example, when the image is the first frame in a set of images, a semantic segmentation model is used to detect the image to determine the first region; or, when the image is a frame other than the first frame in a set of images, the first region of the image is determined based on the position and extent of the first region in the first frame of the set of images. This improves the speed of extracting the first region from image frames other than the first frame.

[0062] Figure 3 is a schematic diagram of a first region extraction method according to an embodiment of this application, used to implement operation 101. As shown in Figure 3, the first region extraction method includes:

[0063] Operation 301: Determine if the image is the first frame in a set of images;

[0064] Operation 302: When the image is the first frame in a set of images, use a semantic segmentation model to detect the image to determine the first region;

[0065] Operation 303: When the image is an image frame other than the first frame in a set of images, determine the first region of the image based on the position and range of the first region of the first image frame in the set of images.

[0066] For example, multiple frames of images taken from the same surveillance camera at the same angle constitute a set of images, and each frame in the set can be labeled with the same group of tags (e.g., ID). If the image sequence comes from two surveillance cameras or from two different shooting angles of the same surveillance camera, it is considered as two sets of images, where each frame in the first set can be labeled with group 0, and each frame in the second set can be labeled with group 1.

[0067] The first region extraction method of this application is used to process the two sets of image sequences sequentially. When the first frame image of the first set of images (i.e., the first image with group label 0) is detected, it is determined that the image is the first frame in the set of images. Therefore, a semantic segmentation model is used to detect the image to determine the first region in the image and store the position coordinate information of the first region, such as the center and range of the first region, or the position coordinates of each vertex of the first region (e.g., the four vertices of a rectangular region). When other frames (e.g., the second frame) in the first set of images are detected (e.g., the Kth image with group label 0, where K is a natural number greater than or equal to 2), since the group label of the other frame image is the same as the group label of the first frame image, it is determined that the image is an image frame other than the first frame in the set of images. Therefore, the first region of the current image is determined according to the position coordinate information of the first region of the first frame image in the set of images that has been stored.

[0068] The following section explains how to use a semantic segmentation model to determine the first region.

[0069] Figure 4 is a schematic diagram of an image captured by a surveillance camera according to an embodiment of this application; Figure 5 is a schematic diagram of a semantic segmentation mask image according to an embodiment of this application; Figure 6 is a schematic diagram of first region detection in an image captured by a surveillance camera according to an embodiment of this application; and Figure 7 is a schematic diagram of the extracted first region according to an embodiment of this application.

[0070] In some examples, a semantic segmentation model is used to semantically segment image 400 shown in Figure 4, resulting in a masked image 500 as shown in Figure 5. For instance, this semantic segmentation model includes a Unified Perceptual Parsing for Scene Understanding (UperNet) model, in which a Small Swin Transformer (Swin-S) network replaces its backbone architecture; furthermore, this semantic segmentation model can be trained using multiple images as training data. The semantic segmentation model defines, for example, 18 object categories, including road, sidewalk, traffic sign, vegetation, sky, car, dotline, and solidline. In the masked image 500 shown in Figure 5, the region corresponding to the object category "road" is region 501.

[0071] In this application, the bounding box of the road region is obtained based on the outline of region 501, such as the dashed frame 601 in Figure 6. Based on the road image region within the dashed frame 601, the aspect ratio of the dashed frame 601 is adjusted, and the region within the adjusted dashed frame 601 is taken as the first region, i.e., the first region 700 as shown in Figure 7. The image of the first region 700 is the output of operation 101, and this image of the first region 700 is input to operation 102 of Figure 1 or to operation 201 of Figure 2.

[0072] In this application, compared with the original image captured by the input surveillance camera, the input extracted first area image can improve the accuracy and speed of road damage detection.

[0073] In operation 201, the preprocessing performed on the first region includes, for example, image flipping (e.g., horizontal flipping and / or vertical flipping), and / or image rotation, and / or resolution adjustment (e.g., increasing or decreasing resolution), and other types of preprocessing may also be performed. Through operation 201, multiple preprocessed images corresponding to the first region are obtained.

[0074] In operation 202, the multiple preprocessed images obtained in operation 201 are input into multiple detection models, and each detection model outputs the corresponding detection result based on the input preprocessed image.

[0075] For example, for a first region, after two or more preprocessing operations 201, n preprocessed images are generated (where n is a natural number greater than or equal to 2); for each of the m detection models (where m is a natural number greater than or equal to 2), the n preprocessed images are input respectively to generate n detection results. Therefore, for m detection models, m*n detection results are generated.

[0076] Furthermore, for operation 102, the first region obtained in operation 101 is input into multiple detection models, and each detection model outputs the corresponding detection result based on the input preprocessed image frame.

[0077] For example, if an image of a first region is input into each of m detection models (where m is a natural number greater than or equal to 2), each of the m detection models will generate one detection result, that is, the m detection models will generate a total of m detection results.

[0078] In operation 102 or operation 202, each detection model (e.g., each of m detection models) can be a detection model based on a deep learning model, such as a detection model trained on a predetermined number of training images. The detection results output by each detection model can, for example, include a list of detection boxes, where the probability of road damage existing in the region represented by the detection box is higher than a threshold; that is, the region represented by the detection box is a region with a high probability of having damage.

[0079] Multiple detection results output by operation 102 (e.g., m detection results) or multiple detection results output by operation 202 (e.g., m*n detection results) can be input into step 103 for processing to generate detection results reflecting damage on the road.

[0080] Figure 8 is a schematic diagram of a post-processing method according to an embodiment of this application. As shown in Figure 8, in at least one embodiment, the processing in step 103 includes:

[0081] Operation 801: Averaging the detection results from multiple detection models; and

[0082] Operation 802: Perform non-maximum suppression (NMS) processing on the averaged results to generate detection results that reflect the damage on the road.

[0083] Figure 9 is a schematic diagram of the road damage detection process according to an embodiment of this application, used to illustrate operations 201, 202 and 103 of Figure 2.

[0084] Taking Figure 9 as an example, the first region in a frame image is preprocessed with image flipping and resolution enhancement to obtain four preprocessed images, namely, preprocessed image 1, preprocessed image 2, preprocessed image 3 and preprocessed image 4, which can be left-right flipped images and images with three different resolutions, respectively.

[0085] Four preprocessed images are input into detection model 1, detection model 2, and detection model 3. At least one of detection model 1, detection model 2, and detection model 3 can be a deep learning model. For example, the deep learning model is based on the YOLO (You Only Look Once) model and is trained through various combinations of parameters or data. Each model is suitable for road damage detection of different sizes and / or different angles.

[0086] For example, during the training process, detection model 1 uses a dataset of road damage images with smaller dimensions and is configured with training parameters for detecting smaller targets to train a detection model capable of detecting road damage with smaller dimensions; during the training process, detection model 2 uses a dataset of road damage images with larger shooting angles and is configured with training parameters for detecting targets with larger shooting angles to train a detection model capable of detecting road damage with larger shooting angles; during the training process, detection model 3 uses a dataset of road damage images with medium or larger dimensions and is configured with training parameters for detecting medium or larger targets to train a detection model capable of detecting road damage with medium or larger dimensions.

[0087] Four preprocessed images are input into detection model 1, resulting in a list of detection boxes for each preprocessed image. These lists are then merged to obtain detection box list 1. Similarly, four preprocessed images are input into detection model 2, resulting in a list of detection boxes for each preprocessed image. These lists are then merged to obtain detection box list 2. Finally, four preprocessed images are input into detection model 3, resulting in a list of detection boxes for each preprocessed image. These lists are then merged to obtain detection box list 3. Each detection box list is, for example, a matrix containing the coordinates of the detection boxes and the detection probabilities of the corresponding categories. The number of detection boxes and categories output by each detection model (e.g., detection model 1, detection model 2, and detection model 3) can be the same, therefore the dimensionality of each list is the same.

[0088] Averaging is performed on multiple lists of detection boxes (e.g., list 1, list 2, and list 3) to merge multiple detection boxes in each list.

[0089] In some examples, detection models 1, 2, and 3 each output the position coordinates of three detection boxes. The position coordinates of each detection box are represented by t, which can be a vector. For example, t has 5 elements, representing the positions of the four vertices and the center point of the detection box (e.g., the detection box is a rectangle). Detection models 1, 2, and 3 also output the probability 1 of each detection box being identified as a road damage area and the probability 2 of being identified as a non-road damage area. These two probability values ​​are combined into a vector (i.e., the class probability vector), which is represented by p. That is, the bounding box tensor of the detection box list 1 output by detection model 1 is [t1,t2,t3], and the class probability is [p1,p2,p3], where t1,t2,t3 represent the position coordinates of the three detection boxes, and p1,p2,p3 represent the class probability vectors of the three detection boxes respectively; the bounding box tensor of the detection box list 2 output by detection model 2 is [t4,t5,t6], and the class probability is [p4,p5,p6], where t4,t5,t6 represent the position coordinates of the three detection boxes, and p4,p5,p6 represent the class probability vectors of the three detection boxes respectively; the bounding box tensor of the detection box list 3 output by detection model 3 is [t7,t8,t9], and the class probability is [p7,p8,p9], where t7,t8,t9 represent the position coordinates of the three detection boxes, and p7,p8,p9 represent the class probability vectors of the three detection boxes respectively. After averaging by 901, the results are as follows: averaged bounding box coordinates, i.e., the first bounding box coordinates are (t1+t4+t7) / 3, the second bounding box coordinates are (t2+t5+t8) / 3, and the third bounding box coordinates are (t3+t6+t9) / 3; and averaged class probabilities, i.e., the class probability vector of the first bounding box is (p1+p4+p7) / 3, the class probability vector of the second bounding box is (p2+p5+p8) / 3, and the class probability vector of the third bounding box is (p3+p6+p9) / 3.

[0090] As shown in Figure 9, after averaging 901, non-maximum suppression (NMS) 902 is applied to the result of averaging 901. That is, for multiple detection boxes with large overlapping areas (e.g., the overlapping area of ​​the detection boxes is greater than the threshold), only the detection boxes with higher probabilities (e.g., the probability is higher than the threshold) are retained, and the detection boxes with lower probabilities (e.g., the probability is lower than the threshold) are deleted.

[0091] For example, after averaging 901, the overlapping area of ​​detection boxes 1, 2 and 3 exceeds a preset threshold. The probability of road damage corresponding to detection box 1 is greater than that corresponding to detection box 2, which is greater than that corresponding to detection box 3. Therefore, during the non-maximum suppression (NMS) process 902, detection box 1 is retained, and detection boxes 2 and 3 are deleted.

[0092] Due to the lack of road damage samples from the perspective of surveillance cameras, it is difficult to verify the accuracy of the road damage detection model in the embodiments of this application. An embodiment of the first aspect of this application also provides a method for synthesizing road damage samples from the perspective of surveillance cameras, using the synthesized images to verify the performance of the road damage detection model. This method for synthesizing road damage samples is not limited to synthesizing road damage samples from the perspective of surveillance cameras, but can also be applied to synthesizing road damage samples from other perspectives, such as patrol vehicles, which detect road damage.

[0093] Figure 10 is a schematic diagram of a method for synthesizing road damage samples according to an embodiment of this application. As shown in Figure 10, the method for synthesizing road damage samples includes:

[0094] Operation 1001: Extract road damage graphics from sample images containing road damage;

[0095] Operation 1002: Overlay the road damage graphic onto a predetermined position in the background image to generate a composite image.

[0096] In this application, as shown in Figure 10, the method for synthesizing road damage samples may further include:

[0097] Operation 1003: Perform at least one of the following processing on the road damage graphic: zoom in, zoom out, adjust contrast, and adjust brightness.

[0098] In operation 1003, during operation 1002, the road damage image processed by operation 1003 is superimposed onto a predetermined position of the background image to generate a composite image.

[0099] The method shown in Figure 10 generates synthetic samples (e.g., synthetic images) that can be used to verify the accuracy of the road damage detection model (e.g., the detection model used in operation 102 or operation 202). In this way, even when the number of real road damage samples is small, the accuracy of the road damage detection model can be effectively verified, which is beneficial for the evaluation and training of the road damage detection model.

[0100] Figure 11 is a schematic diagram of a road damage image. Figure 12 is a schematic diagram of the road damage graphic annotation in Figure 11. Figure 13 is a schematic diagram of a road damage sample synthesized based on the road damage graphic annotation in Figure 12. Figure 14 is another schematic diagram of a road damage image. Figure 15 is a schematic diagram of the road damage graphic annotation in Figure 14. Figure 16 is a schematic diagram of a road damage sample synthesized based on the road damage graphic annotation in Figure 15.

[0101] For example, as shown in Figures 11, 12, and 13, this embodiment of the application uses a sample generation tool. The road damage image shown in Figure 11, the road damage graphic annotation information shown in Figure 12, and the road image captured by the surveillance camera (i.e., the background image) are input into the sample generation tool. Clicking on the background image at the location where the road damage graphic needs to be pasted, the road damage graphic from Figure 11 is pasted to the specified location in the background image, thereby synthesizing the road damage sample from the surveillance camera's perspective shown in Figure 13. A scaling factor can be set to enlarge or reduce the size of the road damage graphic in Figure 11. The brightness and contrast of the road damage image (as shown in Figure 11) can also be adjusted to make the road damage graphic in Figure 11 blend more closely into the background image, making the synthesized road damage sample (as shown in Figure 13) visually more natural.

[0102] For example, as shown in Figures 14, 15, and 16, the road damage image shown in Figure 14, the road damage graphic annotation information shown in Figure 15, and the road image captured by the surveillance camera (i.e., the background image) are input into the sample generation tool. Clicking on the background image at the location where the road damage graphic needs to be pasted will paste the road damage graphic from Figure 14 into the specified location in the background image, thus synthesizing the road damage sample from the surveillance camera's perspective shown in Figure 16. A scaling factor can be set to enlarge or reduce the size of the road damage graphic in Figure 14. The brightness and contrast of the road damage image (as shown in Figure 14) can also be adjusted to make the road damage graphic in Figure 14 blend more closely into the background image, making the synthesized road damage sample (as shown in Figure 16) appear more visually natural.

[0103] Images of road damage (as shown in Figures 11 and 14) can, for example, come from a well-annotated public road damage dataset, such as the Road Damage Dataset 2020 (RDD-2020). This dataset is captured by cameras mounted on vehicles and includes, for example, four types of road damage: longitudinal cracks, transverse cracks, alligator cracks, and potholes.

[0104] In this application, road damage images from a well-annotated public road damage dataset (as shown in Figures 11 and 14) can be used to train a road damage detection model (e.g., the detection model used in operation 102 or operation 202). Multiple (e.g., 168) synthetic samples of different types of road damage, synthesized using the road damage sample synthesis method shown in Figure 10, are pasted to validate the accuracy of the road damage detection model.

[0105] Figure 17 is a schematic diagram of actual on-site images captured by a surveillance camera. These on-site images can form a validation dataset 1, for example, validation dataset 1 includes 641 images. Validation dataset 1 contains a small number of images with road damage (positive samples), for example, 61, and a larger number of images without road damage (negative samples), namely 580.

[0106] Figure 18 is a schematic diagram of a road damage sample synthesized using the method of this application embodiment. The synthesized road damage samples can form a verification dataset 2, which for example includes 168 images. All images in the verification dataset 2 contain road damage (positive samples), and there are no images that do not contain road damage (negative samples).

[0107] Table 1 is a set of data on the verification results of the road damage detection model in the embodiments of this application.

[0108] In this embodiment of the application, the above-mentioned verification dataset 1 and verification dataset 2 are used to verify the accuracy of the road damage detection model. The detection results of the two inputs, the original image (e.g., image 400 shown in FIG4) and the extracted first region (e.g., the first region 700 shown in FIG7), are compared, as shown in Table 1.

[0109] Table 1

[0110] In Table 1, FP represents the number of true negative samples in an image detected as a positive sample; FN represents the number of true positive samples in an image detected as a negative sample; TP represents the number of true positive samples in an image detected as a positive sample; and Recall represents the proportion of true positive samples that were successfully detected. Calculated; Precision represents the proportion of true positive samples in an image detected as positive, derived from... Calculated; F1-Score represents the harmonic average based on Recall and Precision, derived from... Calculated.

[0111] The results show that for validation dataset 1 (a dataset of real-world images captured by surveillance cameras), the method of extracting road images and inputting them into the detection model according to this embodiment improves the F1-Score by 24%. For validation dataset 2 (a dataset of road damage sample images synthesized using the road damage sample synthesis method of this embodiment), the method of extracting road images and inputting them into the detection model according to this embodiment improves the F1-Score by 3%. This demonstrates that the method of extracting road images (e.g., extracting the first region) according to this application can effectively improve detection accuracy.

[0112] The above description only covers the steps or processes related to this application, but this application is not limited thereto. The method for detecting the quantity of items may also include other steps or processes; for details of these steps or processes, please refer to the prior art. Furthermore, the above description only uses some structural examples of the model used in the method for detecting the quantity of items to illustrate the embodiments of this application, but this application is not limited to these structures, and appropriate modifications can be made to these structures. All such modifications should be included within the scope of the embodiments of this application.

[0113] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0114] As can be seen from the above embodiments, according to the embodiments of the first aspect of this application, in the road damage detection method, multiple detection models are used to detect road damage in the extracted road area image. The multiple detection models can detect road damage of different shapes and / or different sizes, thereby improving the accuracy of road damage detection.

[0115] Second aspect of the embodiments

[0116] This application provides a road damage detection device, corresponding to the road damage detection method of the first aspect embodiment. The second aspect embodiment is identical to the first aspect embodiment and will not be described again.

[0117] Figure 19 is a schematic diagram of a road damage detection device according to an embodiment of this application. As shown in Figure 19, the road damage detection device 1900 includes: an extraction device 1901, a detection device 1902, and a post-processing device 1903.

[0118] Extraction device 1901 extracts a first region containing a road from an image, the area of ​​which is smaller than the area of ​​the image;

[0119] The detection device 1902 inputs the first region into multiple detection models within the device, wherein each detection model outputs a detection result for the first region; and

[0120] The post-processing unit 1903 processes the detection results output by each of the multiple detection models to generate detection results reflecting the damage on the road.

[0121] In at least one embodiment, the road damage detection device 1900 further includes:

[0122] The preprocessing unit 1904 performs two or more preprocessing operations on the first region to generate multiple preprocessed images.

[0123] in,

[0124] The multi-frame preprocessed images are respectively input into each of the multiple detection models in the detection device, wherein each detection model outputs a detection result for each frame of preprocessed images.

[0125] In at least one embodiment, when the image is the first frame in a set of images, the extraction device 1901 uses a semantic segmentation model to detect the image to determine a first region; or...

[0126] When the image is an image frame other than the first frame in a set of images, the extraction device 1901 determines the first region of the image based on the position and extent of the first region of the first image frame in the set of images.

[0127] In at least one embodiment, the post-processing device 1903 includes:

[0128] The detection results from multiple detection models are averaged; and

[0129] The averaged results are subjected to non-maximum suppression (NMS) to generate detection results that reflect the damage on the road.

[0130] In at least one embodiment, the detection results output by the detection model include a list of detection boxes, wherein the probability of road damage existing in the area represented by the detection boxes is higher than a threshold.

[0131] In at least one embodiment, the road damage detection device 1900 further includes a sample generation device 1905.

[0132] Sample generation device 1905:

[0133] Extract road damage graphics from sample images containing road damage; and

[0134] The road damage graphic is overlaid onto a predetermined position in the background image to generate a composite image.

[0135] In at least one embodiment, synthesized images are used as verification data to verify the performance of the detection model.

[0136] In at least one embodiment, the sample generation device 1905 performs at least one process on the road damage graphic, including magnification, reduction, contrast adjustment, and brightness adjustment.

[0137] In this process, the processed image of road damage is superimposed onto a predetermined position in the background image to generate a composite image.

[0138] In at least one embodiment, the image is an image from a traffic surveillance video.

[0139] It is worth noting that the above description only covers the components or devices relevant to this application, but this application is not limited thereto. The road damage detection device 1900 may also include other components or devices, and for details regarding these components or devices, please refer to related technologies.

[0140] For simplicity, Figure 19 only illustrates the connection relationships or signal flow between the various components or devices, but those skilled in the art should understand that various related technologies such as bus connections can be used. The aforementioned components or devices can be implemented using hardware facilities such as processors and memory; the embodiments of this application do not limit this.

[0141] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0142] Third aspect of the embodiments

[0143] This application provides an electronic device including a road damage detection device 1900 as described in the second aspect of the embodiment, the contents of which are incorporated herein by reference. This electronic device may be, for example, a computer, server, workstation, laptop computer, smartphone, etc.; however, this application is not limited thereto.

[0144] Figure 20 is a schematic diagram of an electronic device according to an embodiment of this application. As shown in Figure 20, the electronic device 2000 may include: a processor (e.g., a central processing unit, CPU) 2010 and a memory 2020; the memory 2020 is coupled to the central processing unit 2010. The memory 2020 can store various data; in addition, it also stores an information processing program 2021, and executes the program 2021 under the control of the processor 2010.

[0145] In some embodiments, the functionality of the road damage detection device 1900 is integrated into the processor 2010. The processor 2010 is configured to implement the road damage detection method as described in the embodiment of the first aspect.

[0146] In some embodiments, the road damage detection device 1900 is configured separately from the processor 2010. For example, the road damage detection device 1900 can be configured as a chip connected to the processor 2010, and the functions of the road damage detection device 1900 can be implemented through the control of the processor 2010.

[0147] Furthermore, as shown in Figure 20, the electronic device 2000 may also include: an input / output (I / O) device 2030 and a display 2040, etc.; the functions of the above components are similar to those in the prior art, and will not be described in detail here. It is worth noting that the electronic device 2000 does not necessarily need to include all the components shown in Figure 20; in addition, the electronic device 2000 may also include components not shown in Figure 20, which can be referred to in related technologies.

[0148] This application also provides a computer-readable program, wherein when the program is executed in an electronic device, the program causes the computer in the electronic device to perform the road damage detection method as described in the first aspect embodiment.

[0149] This application also provides a storage medium storing a computer-readable program, wherein the computer-readable program causes a computer in an electronic device to perform the road damage detection method as described in the first aspect embodiment.

[0150] The apparatus and methods described above in this application can be implemented in hardware or in combination with software. This application relates to a computer-readable program that, when executed by a logic component, enables the logic component to implement the apparatus or components described above, or to implement the various methods or steps described above. This application also relates to storage media for storing the above programs, such as hard disks, magnetic disks, optical disks, DVDs, flash memory, etc.

[0151] The methods / apparatus described in conjunction with the embodiments of this application can be directly embodied in hardware, software modules executed by a processor, or a combination of both. For example, one or more and / or combinations of one or more functional block diagrams shown in the figures can correspond to various software modules in a computer program flow, or to various hardware modules. These software modules can correspond to the various steps shown in the figures, respectively. These hardware modules can be implemented, for example, using a field-programmable gate array (FPGA) to embed these software modules.

[0152] The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium can reside in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a high-capacity MEGA-SIM card or a high-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the high-capacity flash memory device.

[0153] One or more and / or one or more combinations of functional blocks described in the accompanying drawings can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described herein. One or more and / or one or more combinations of functional blocks described in the accompanying drawings can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0154] The present application has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles thereof, and these modifications and variations are also within the scope of the present application.

Claims

1. A road damage detection device, characterized in that, The device includes: An extraction device extracts a first region containing a road from an image, the area of ​​which is smaller than the area of ​​the image. The detection device inputs the first region into multiple detection models, wherein each detection model outputs a detection result for the first region; and The post-processing device processes the detection results output by each of the multiple detection models to generate detection results reflecting the damage on the road.

2. The apparatus as claimed in claim 1, characterized in that, The device further includes: The preprocessing unit performs two or more preprocessing operations on the first region to generate multiple preprocessed images. in, The multi-frame preprocessed images are respectively input into each of the multiple detection models in the detection device, wherein each detection model outputs a detection result for each frame of preprocessed images.

3. The apparatus as described in claim 1, characterized in that, The post-processing device: The detection results from the multiple detection models are averaged; and The results of the averaging process are subjected to non-maximum suppression (NMS) to generate detection results that reflect the damage on the road.

4. The apparatus as claimed in claim 1, characterized in that, The detection model outputs a list of detection boxes, where the probability of road damage in the area represented by each detection box is higher than a threshold.

5. The apparatus as claimed in claim 1, characterized in that, When the image is the first frame in a set of images, the extraction device uses a semantic segmentation model to detect the image in order to determine the first region; or, When the image is an image frame other than the first frame in a set of images, the extraction device determines the first region of the image based on the position and range of the first region of the first image frame in the set of images.

6. The apparatus as claimed in claim 1, characterized in that, The device also includes a sample generation device. The sample generation device: Extract road damage graphics from sample images containing road damage; and The road damage graphic is superimposed onto a predetermined position in the background image to generate a composite image.

7. The apparatus as claimed in claim 6, characterized in that, The synthesized images are used as verification data to validate the performance of the detection model.

8. The apparatus as claimed in claim 6, characterized in that, The sample generation device performs at least one of the following processes on the road damage graphic: magnification, reduction, contrast adjustment, and brightness adjustment. The processed road damage image is superimposed onto a predetermined position of a background image to generate the synthesized image.

9. The apparatus as claimed in claim 1, characterized in that, The image is from a traffic surveillance video.

10. An electronic device having a road damage detection device as claimed in any one of claims 1 to 9.

11. A method for detecting road damage, characterized in that, The method includes: Extract a first region containing roads from the image, the area of ​​the first region being smaller than the area of ​​the image; The first region is input into multiple detection models, and each detection model outputs a detection result for the first region; and The detection results output by each of the multiple detection models are processed to generate detection results reflecting the damage on the road.

12. The method as described in claim 11, characterized in that, The method further includes: Perform two or more preprocessing steps on the first region to generate multiple preprocessed images. in, The preprocessed images are input into each of the multiple detection models, and each detection model outputs the detection result for each preprocessed image.

13. The method as described in claim 11, characterized in that, The process includes: The detection results from the multiple detection models are averaged; and The results of the averaging process are subjected to non-maximum suppression (NMS) to generate detection results that reflect the damage on the road.

14. The method as described in claim 11, characterized in that, The detection model outputs a list of detection boxes, where the probability of road damage in the area represented by each detection box is higher than a threshold.

15. The method as described in claim 11, characterized in that, When the image is the first frame in a set of images, a semantic segmentation model is used to detect the image to determine the first region; or, When the image is an image frame other than the first frame in a set of images, the first region of the image is determined according to the position and extent of the first region of the first image frame in the set of images.

16. The method as described in claim 11, characterized in that, The method further includes: The performance of the detection model is verified using synthesized images as validation data.

17. The method as described in claim 16, characterized in that, The method further includes: Extract road damage graphics from sample images containing road damage; and The road damage graphic is superimposed onto a predetermined position in the background image to generate the synthesized image.

18. The method as described in claim 17, characterized in that, The method further includes: The road damage graphic is subjected to at least one of the following processing methods: magnification, reduction, contrast adjustment, and brightness adjustment. The processed road damage image is superimposed onto a predetermined position of a background image to generate the synthesized image.

19. The method as described in claim 11, characterized in that, The image is from a traffic surveillance video.

20. A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor coupled to the storage medium to perform the road damage detection method as described in any one of claims 11 to 19.

Images