Method and apparatus for determining road surface condition by using plurality of models

Abstract

The present invention relates to a method and apparatus for determining a road surface condition by using a plurality of models, and the method for determining a road surface condition, according to an embodiment of the present invention, comprises the steps of: localizing a potential dangerous area by detecting a potential dangerous object from a road surface image of a road on the basis of an object detection model; identifying a potential dangerous object on a pixel-by-pixel basis in the localized potential dangerous area by using a semantic segmentation model; and determining a road surface condition by analyzing the risk degree of the identified potential dangerous object.

Description

Method and apparatus for determining road surface conditions using multiple models

[0001] The present invention relates to a road surface condition determination technology, and more specifically, to a road surface condition determination method and apparatus using multiple models to accurately determine the road surface condition by utilizing multiple artificial intelligence models.

[0002] Poor road conditions refer to a phenomenon where road surface performance rapidly deteriorates due to aging and climate change, leading to defects such as potholes and cracks caused by continuous vehicle loads. Since these road defects occur in irregular shapes and random locations, drivers are exposed to unexpected dangers, increasing the likelihood of various accidents.

[0003] A Pavement Monitoring System (PMS) can be implemented using an Artificial Intelligence (AI) model to prevent such accidents.

[0004] Conceptually, the use of a semantic segmentation model among AI models is suitable for road analysis. However, road surface condition determination devices face problems when analyzing road surfaces based on semantic segmentation models. Since segmentation models generally require a decoding process, the amount of computation is considerably large. Furthermore, it may be difficult to ensure stable execution speeds on edge devices with segmentation models. Additionally, if the model input size is reduced, information regarding the unique characteristics of cracks may be lost.

[0005] Meanwhile, when developing an AI model for a road surface condition determination device, there is a difficulty in constructing a crack dataset. In the case of cracks, the dataset format can be of the following two forms. First, binary GT images may make up about 90% or more. Second, tree type data structures may make up about 10% or less.

[0006] Most conventional crack datasets are very difficult to utilize from the perspective of road surface condition assessment devices. Conventional crack datasets differ significantly from images acquired by moving vehicles in terms of shooting angle, field of view (FoV), and clarity. Furthermore, due to the surface texture characteristics of asphalt, the acquisition of automated binary GT images is impossible. In this context, while a realistic annotation method involves polygon editing, it is unsuitable for constructing crack data.

[0007] Meanwhile, road surface condition assessment devices face challenges when analyzing road surfaces based on object detection models. Although constructing crack datasets is easy, unique crack information may be lost when resizing or scaling is changed, potentially leading to a significant decrease in detection accuracy. Furthermore, such devices may fail to provide sufficient information for precise and diverse road analyses. For instance, a road surface condition assessment device may not be able to provide information regarding the extent to which lane markings have worn away or the size of potholes.

[0008] Embodiments of the present invention aim to provide a method and apparatus for determining road surface conditions using multiple models, for accurately determining road surface conditions by utilizing multiple artificial intelligence models.

[0009] However, the problem to be solved by the present invention is not limited thereto and may be extended in various ways within an environment that does not deviate from the spirit and scope of the present invention.

[0010] According to one embodiment of the present invention, a road surface condition determination method performed by a road surface condition determination device may be provided, comprising: a step of detecting potential hazardous objects based on an object detection model in a road surface image to localize a potential hazardous area; a step of identifying potential hazardous objects in pixel units using a semantic segmentation model in the localized potential hazardous area; and a step of determining the road surface condition by analyzing the risk level of the identified potential hazardous objects.

[0011] The above method may further include the step of outputting at least one of the object information of the identified potential risk object, the risk level of the analyzed potential risk object, and the determined road surface condition.

[0012] The step of localizing the potential risk area described above may involve detecting a first class area by performing object detection based on an object detection model in the road surface image, and generating first object data for the detected first class area.

[0013] The step of identifying the potential risk object may involve performing image segmentation based on a segmentation model in the localized potential risk area to detect a second class area, and label clustering the detected second class area to generate second object data.

[0014] The above method may further include the step of tracking the detected potential risk object using an additional tracker after the potential risk object is detected.

[0015] The step of determining the road surface condition above can analyze the risk level of the potential risk object using the number of pixels or the bounding box area of ​​the identified potential risk object.

[0016] The step of determining the road surface condition above is,

[0017]

[0018] Here, ratio is the risk ratio, c 픽셀 The number of pixels in the crack, p 영역 ε represents the area of ​​the pothole, w represents the weight of the pothole, and

[0019] Based on the risk ratio of the above mathematical formula, the risk level of the above potential risk object can be analyzed as one of a normal state, a warning state, or a faulty state.

[0020] Meanwhile, according to another embodiment of the present invention, a road surface condition determination device using multiple models may be provided, comprising: a memory for storing one or more programs; and a processor for executing the one or more stored programs, wherein the processor detects potential hazardous objects based on an object detection model in a road surface image to localize a potential hazardous area, identifies potential hazardous objects in pixel units using a semantic segmentation model in the localized potential hazardous area, and determines the road surface condition by analyzing the risk level of the identified potential hazardous objects.

[0021] The processor can output at least one of the object information of the identified potential risk object, the risk level of the analyzed potential risk object, and the determined road surface condition.

[0022] The processor can detect a first class region by performing object detection based on an object detection model in the road surface image and generate first object data for the detected first class region.

[0023] The processor can detect a second class region by performing image segmentation based on a segmentation model in the localized potential risk region, and generate second object data by label clustering the detected second class region.

[0024] After the potential risk object is detected, the processor can additionally use a tracker to track the detected potential risk object.

[0025] The processor can analyze the risk level of the potential risk object using the number of pixels or the bounding box area of ​​the identified potential risk object.

[0026] The above processor is,

[0027]

[0028] Here, ratio is the risk ratio, c 픽셀 The number of pixels in the crack, p 영역 ε represents the area of ​​the pothole, w represents the weight of the pothole, and

[0029] Based on the risk ratio of the above mathematical formula, the risk level of the above potential risk object can be analyzed as one of a normal state, a warning state, or a faulty state.

[0030] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0031] Embodiments of the present invention can accurately determine road surface conditions by utilizing a plurality of artificial intelligence models.

[0032] Embodiments of the present invention can provide an algorithm for efficiently determining the road surface condition through a plurality of model-based inference operations adjusted to overcome the limited performance of an edge device installed in a vehicle.

[0033] Embodiments of the present invention can reduce computational load while maintaining detection performance by sequentially utilizing multiple pre-trained models (object detection model and semantic segmentation model).

[0034] Embodiments of the present invention can intensively detect a first object, such as a crack, by inputting a full-scale image of a valid region of interest cropped from a road surface image into an object detection model.

[0035] Embodiments of the present invention input a region of interest image downscaled from a road surface image into an image segmentation model to secure realistic computational cost and performance, and can intensively detect precise areas of second objects such as roadmarks, manholes, patches, and potholes.

[0036] FIG. 1 is a flowchart illustrating a method for determining road surface conditions using multiple models according to an embodiment of the present invention.

[0037] FIG. 2 is a diagram showing an example of actual object detection based on an object detection model in a road surface condition determination method according to an embodiment of the present invention.

[0038] FIG. 3 is a diagram showing the structure of an object detection model used in a road surface condition determination method according to one embodiment of the present invention.

[0039] FIG. 4 is a flowchart illustrating the operation of generating first object data based on an object detection model in a road surface determination method according to an embodiment of the present invention.

[0040] FIG. 5 is a diagram showing an example of a cropping operation of a first region of interest image for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0041] FIG. 6 is a diagram showing an example of a first class area detection operation based on an object detection model for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0042] FIG. 7 is a diagram showing an example of a post-processing operation for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0043] FIG. 8 is a diagram showing an example of a risk calculation operation based on an object analysis model for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0044] FIG. 9 is a diagram showing an example of identifying a crack area based on a semantic segmentation model in a road surface condition determination method according to an embodiment of the present invention.

[0045] FIG. 10 is a diagram showing the structure of a semantic segmentation model used in a road surface condition determination method according to one embodiment of the present invention.

[0046] FIG. 11 is a diagram showing an example of identifying potential hazardous objects on a road using a plurality of models when a road surface condition determination method according to an embodiment of the present invention is performed at an edge device.

[0047] FIG. 12 is a flowchart illustrating the operation of generating second object data based on a segmentation model in a road surface determination method according to an embodiment of the present invention.

[0048] FIG. 13 is a diagram showing an example of a cropping operation of a second region of interest image for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0049] FIG. 14 is a diagram showing an example of a scale adjustment operation for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0050] FIG. 15 is a diagram showing an example of an image segmentation operation based on a segmentation model for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0051] FIG. 16 is a diagram showing an example of a label clustering operation for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0052] FIG. 17 is a diagram showing an example of a noise filtering operation for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0053] FIGS. 18 and 19 are drawings illustrating examples of a custom U-net model for an operation to generate second object data in a road surface determination method according to an embodiment of the present invention.

[0054] FIG. 20 is a configuration diagram of a road surface condition determination device using multiple models according to one embodiment of the present invention.

[0055] The present invention is capable of various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the technical spirit and scope of the present invention. In describing the present invention, detailed descriptions of related prior art are omitted if it is determined that such detailed descriptions may obscure the essence of the present invention.

[0056] Terms such as "first," "second," etc., may be used to describe various components, but the components are not limited by these terms. The terms are used solely for the purpose of distinguishing one component from another.

[0057] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. While the terms used in this invention have been selected to be as widely used as possible in consideration of their functions within the invention, they may vary depending on the intent of those skilled in the art, case law, or the emergence of new technologies. Furthermore, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0058] A singular expression includes a plural expression unless the context clearly indicates otherwise. In the present invention, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0059] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing with reference to the accompanying drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0060] FIG. 1 is a flowchart illustrating a method for determining road surface conditions using multiple models according to an embodiment of the present invention.

[0061] As illustrated in FIG. 1, in step S101, the road surface condition determination device receives an image of the road surface. The road surface condition determination device may acquire an image of the road surface through a camera or sensor installed on a vehicle. The resolution of the road surface image may differ as it is captured under various lighting conditions and weather conditions.

[0062] In step S102, the road surface condition determination device performs an inference operation based on an object detection model in the input road surface image. That is, the road surface condition determination device can detect potential hazardous objects based on the object detection model in the road surface image and localize potential hazardous areas. Here, the road surface condition determination device can detect a first class area by performing object detection based on the object detection model in the road surface image and generate first object data for the detected first class area.

[0063] In step S103, the road surface condition determination device checks whether a potential danger area is localized in the road surface image of the road.

[0064] In step S104, if a potential danger area is localized in the road surface image, the road surface condition determination device performs an inference operation based on a Semantic Segmentation Model in the localized potential danger area. That is, the road surface condition determination device can identify potential danger objects on a pixel-by-pixel basis based on the Semantic Segmentation Model in the localized potential danger area. Here, the road surface condition determination device can detect a second class area by performing image segmentation based on the segmentation model in the localized potential danger area, and generate second object data by label clustering the detected second class area. On the other hand, if a potential danger area is not localized in the road surface image, the road surface condition determination device repeats the process from step S101, where the road surface image is received.

[0065] In step S105, the road surface condition determination device checks whether potential risk objects are identified in pixel units within a localized potential risk area.

[0066] In step S106, if a potential risk object is identified in pixel units within a localized potential risk area, the road surface condition determination device analyzes the risk level of the identified potential risk object to determine the road surface condition. Conversely, if a potential risk object is not identified in pixel units within a localized potential risk area, the road surface condition determination device repeats the process starting from step S101, which receives the road surface image.

[0067] Here, the road surface condition determination device can analyze the risk level of a potential risk object using the number of pixels or the bounding box area of ​​the identified potential risk object.

[0068] According to the embodiments, the road surface condition determination device can analyze the risk level of a potential risk object into one of a normal state, a warning state, and a bad state according to the risk ratio of [Equation 1] below.

[0069]

[0070] Here, ratio is the risk ratio, c 픽셀 The number of pixels in the crack, p 영역 represents the area of ​​the pothole, and w represents the weight of the pothole.

[0071] For example, the road surface condition determination device can analyze the risk level of a potential hazardous object as a normal state when the risk ratio calculated according to the risk ratio of [Mathematical Formula 1] is 0 or greater and less than 0.05 (0 <= ratio < 0.05: normal). Next, the road surface condition determination device can analyze the risk level of a potential hazardous object as a warning state when the risk ratio calculated according to the risk ratio of [Mathematical Formula 1] is 0.05 or greater and less than 0.1 (0.05 <= ratio < 0.1: warning). Next, the road surface condition determination device can analyze the risk level of a potential hazardous object as a poor state when the risk ratio calculated according to the risk ratio of [Mathematical Formula 1] is 0.1 or greater (0.1 <= ratio: poor).

[0072] In step S107, the road surface condition determination device outputs road surface information of the determined road. Here, the road surface information may include at least one of object information of an identified potential risk object, a risk level of an analyzed potential risk object, and a determined road surface condition.

[0073] FIG. 2 is a diagram showing an example of actual object detection based on an object detection model in a road surface condition determination method according to an embodiment of the present invention.

[0074] As illustrated in FIG. 2, the road surface condition determination device can detect potential hazardous objects (112) by performing an inference operation based on an object detection model on an input road surface image (111). The potential hazardous object (112) illustrated in FIG. 2 is an alligator crack, which is shown as an example of a potential hazardous object, and is not limited to a specific hazardous object.

[0075] FIG. 3 is a diagram showing the structure of an object detection model used in a road surface condition determination method according to one embodiment of the present invention.

[0076] As illustrated in FIG. 3, a road surface image (111) is input into an object detection model. The object detection model performs an inference operation on the input road surface image (111). Then, potential hazardous objects (112) detected in the road surface image (111) are displayed in the form of bounding boxes.

[0077] According to the embodiments, the object detection model may use the YOLO (You Only Look Once) model as an algorithm for real-time object detection, but other object detection models other than the YOLO model may also be used.

[0078] A road surface condition determination device can detect cracks (e.g., alligator, longitudinal, lateral) or potholes as potential hazardous objects of the road in a road surface image (111) based on an object detection model.

[0079] After a potential hazardous object is detected, the road surface condition determination device can use an additional tracker to continue tracking the detected potential hazardous object.

[0080] FIG. 4 is a flowchart illustrating the operation of generating first object data based on an object detection model in a road surface determination method according to an embodiment of the present invention.

[0081] FIG. 4 is an example of an operation in which a road surface determination device according to one embodiment of the present invention generates first object data to localize a potential danger area.

[0082] According to the embodiments, a road surface determination device can detect a first class area by performing object detection based on an object detection model in a road surface image to localize a potential danger area, and generate first object data for the detected first class area. Hereinafter, an operation embodiment for generating first object data based on an object detection model to localize a potential danger area will be described.

[0083] As illustrated in FIG. 4, in step S201, the road surface condition determination device crops a portion of the road area from the road surface image to extract a first area of ​​interest image.

[0084] In step S202, the road surface condition determination device detects a first class region by performing object detection based on an object detection model in the extracted first region of interest image.

[0085] In step S203, the road surface condition determination device performs post-processing on the detected first class area.

[0086] In step S204, the road surface condition determination device measures the first object information for the post-processed first class area.

[0087] In step S205, the road surface condition determination device calculates the risk level of the first object based on an object analysis model in the first class area where post-processing has been performed.

[0088] In step S206, the road surface condition determination device removes noise from the calculated risk level and generates first object data using the noise-removed risk level and the measured first object information.

[0089] FIG. 5 is a diagram showing an example of a cropping operation of a first region of interest image for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0090] As illustrated in FIG. 5, the road surface condition determination device crops a portion of the road area from the road surface image (210) to extract a first area of ​​interest image (211). The first area of ​​interest image (211) may be different from the second area of ​​interest image (111).

[0091] FIG. 6 is a diagram showing an example of a first class area detection operation based on an object detection model for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0092] The road surface condition determination device detects a first class area (220) by performing object detection based on an object detection model on an extracted first area of ​​interest image (211). Here, the first class area (220) can be output as an object detection list containing at least one first class area. The object detection operation designates the area where the first object is detected in the form of a box. A first class area in the form of a box is detected, and object detection results, such as the class containing the object and confidence, can be output through the object detection model. Confidence is an indicator that indicates how certain the probability is that an object detected by the object detection model (e.g., a crack on the road) belongs to a specific category.

[0093] FIG. 7 is a diagram showing an example of a post-processing operation for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0094] The road surface condition determination device performs post-processing on the detected first class area (220). For example, the road surface condition determination device may perform post-processing operations, such as non-maximum suppression, on the first class area detected in FIG. 6 to remove the remaining box areas while leaving only the box areas that are actually meaningful (i.e., box areas containing first objects such as cracks). For example, through post-processing operations, the eight first class areas (220) shown in FIG. 6 are removed while leaving only the five first class areas (221 to 224) shown in FIG. 7. The five first class areas (221 to 224) can be distinguished according to class color. The orange box is a first class area (221) representing a vertical crack, the light green box is a first class area (222) representing a horizontal crack, the blue box is a first class area (223) representing an alligator crack, and the yellow box is a first class area (224) representing a manhole.

[0095] FIG. 8 is a diagram showing an example of a risk calculation operation based on an object analysis model for generating first object data in a road surface determination method according to an embodiment of the present invention.

[0096] The road surface condition determination device can measure first object information, including object size or location, for a first class area (223) containing an alligator crack. The road surface condition determination device can calculate the risk level of the first object based on the first object information, including size or location, and an object analysis model. In this way, the road surface condition determination device can calculate the risk level of the first object based on an object analysis model using at least one of the first object size, first object location, and first object type from the measured first object information. For example, the road surface condition determination device can quantify the risk rate or nature of the crack by inputting the first object image, the first object class, and the location and size of the first object (e.g., the size and location of the crack estimate image). The road surface condition determination device can accurately calculate the risk level using the crack type or the size and location of the crack image. For example, the road surface condition determination device can calculate the crack risk level as more dangerous as the size of the crack image increases. In addition, the road surface condition determination device can calculate the crack risk as becoming more dangerous in the order of vertical cracks, horizontal cracks, and allicator cracks. In addition, the road surface condition determination device can calculate the crack risk as becoming more dangerous the closer the location of the crack image on the road is to a location where many vehicles travel. For example, if the risk has a risk value of 0 to 9, the risk of the crack shown in FIG. 13 can be calculated as 6.

[0097] The road surface condition determination device removes noise from the calculated risk level and generates first object data using the noise-removed risk level and the measured first object information. The first object data may include not only the first object image but also the first object information and the risk level for the first object. For example, the first object data containing a crack may include a crack image, a crack type, a crack image size, and a crack risk level.

[0098] FIG. 9 is a diagram showing an example of identifying a crack area based on a semantic segmentation model in a road surface condition determination method according to an embodiment of the present invention.

[0099] As illustrated in FIG. 9, the road surface condition determination device can identify potential risk objects (312) in pixel units based on a semantic segmentation model in a localized potential risk area (311). The potential risk object (312) illustrated in FIG. 9 is indicated as a crack as an example of a potential risk object and is not limited to a specific risk object.

[0100] FIG. 10 is a diagram showing the structure of a semantic segmentation model used in a road surface condition determination method according to one embodiment of the present invention.

[0101] According to the embodiments, the semantic segmentation model can be an artificial intelligence model such as Unet or Unet3+, and is not limited to a specific segmentation model. Here, the Unet model has the flexibility to process N-band input images. The input layer of the Unet model can be adjusted to accept N channels, and the filter of the first convolutional layer can process N channels. Through the encoding process, the Unet model can pass the road surface image through a shrinkage path to gradually reduce the spatial dimension and increase the number of channels. Then, through the decoding process, the semantic segmentation model can restore the spatial dimension and reduce the number of channels in the expansion path through upsampling and convolution. Subsequently, the Unet model can perform post-processing operations through FC-CRF (Fully-Connected Conditional Random Field). FC-CRF can refine the output of Unet to provide more accurate boundaries and consistent segmentation. This process can improve the quality of segmentation by considering the spatial and visual relationships between pixels.

[0102] A road surface condition determination device can classify potential road hazards, such as cracks or potholes, at the pixel level based on semantic segmentation models such as Unet.

[0103] FIG. 11 is a diagram showing an example of identifying potential hazardous objects on a road using a plurality of models when a road surface condition determination method according to an embodiment of the present invention is performed at an edge device.

[0104] As illustrated in FIG. 11, the road surface condition determination device can identify potential hazardous objects (412) by performing an object detection model-based inference operation and a semantic segmentation model-based inference operation on an input road surface image (411). The potential hazardous object (412) illustrated in FIG. 11 is indicated as a crack as an example of a potential hazardous object, and is not limited to a specific hazardous object.

[0105] The road surface condition determination device may be included in an edge device installed on a vehicle or may operate as a component of an edge device. To overcome the limited performance of the edge device, the road surface condition determination device may perform inference operations by adjusting multiple artificial intelligence models. Here, the road surface condition determination device may localize the location of potential hazard data of the road over the entire area of ​​the road surface image (411) based on an object detection model. Then, the road surface condition determination device may input the localized potential hazard data into a semantic segmentation model to identify potential hazard objects, such as cracks or potholes, on a pixel-by-pixel basis.

[0106] FIG. 12 is a flowchart illustrating the operation of generating second object data based on a segmentation model in a road surface determination method according to an embodiment of the present invention.

[0107] FIG. 12 is an example of an operation in which a road surface determination device according to one embodiment of the present invention generates second object data to identify a potential danger area.

[0108] According to the embodiments, the road surface determination device can detect a second class area by performing image segmentation based on a segmentation model in a localized potential risk area to identify a potential risk area, and generate second object data by label clustering the detected second class area.

[0109] Hereinafter, an example of operation for generating second object data based on an object detection model for identifying potential risk areas will be described.

[0110] As illustrated in FIG. 12, in step S301, the road surface condition determination device crops a portion of the road area from the road surface image to extract a second area of ​​interest image.

[0111] In step S302, the road surface condition determination device adjusts the scale of the extracted second region of interest image.

[0112] In step S303, the road surface condition determination device performs image segmentation based on a segmentation model on the adjusted second region of interest image to detect the second class region.

[0113] In step S304, the road surface condition determination device clusters labels by object class in the detected second class area.

[0114] In step S305, the road surface condition determination device generates second object data by removing noise from a label clustered label list.

[0115] FIG. 13 is a diagram showing an example of a cropping operation of a second region of interest image for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0116] The road surface condition determination device crops a portion of the road area from the road surface image (510) to extract a second area of ​​interest image (511). Here, the road surface image (510) may be the front image of the vehicle image in front. The road surface condition determination device can generate a second area of ​​interest image (511) by cropping a portion of the road surface area of ​​the vehicle image in front. If a second object, such as a pothole, is found in the entire road surface image, the amount of computation increases. Therefore, the road surface condition determination device can extract a second area of ​​interest image (511) by boxing only the necessary second area of ​​interest from the entire road surface image (510).

[0117] FIG. 14 is a diagram showing an example of a scale adjustment operation for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0118] The road surface condition determination device can generate a scaled second region of interest image (512) by adjusting the scale of the extracted second region of interest image (511). For example, the road surface condition determination device can scale the second region of interest image (511) to 1 / 2 or 1 / 3. By reducing the scale of the second region of interest image (511) in this way, the amount of computation required to perform image segmentation operations is reduced.

[0119] FIG. 15 is a diagram showing an example of an image segmentation operation based on a segmentation model for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0120] The road surface condition determination device detects a second class region (521) by performing image segmentation based on a segmentation model on a scaled second region of interest image (512). Here, the second class region (521) can be extracted through image segmentation. For example, the road surface condition determination device can extract at least one of a patch, manhole, road mark, and pothole by performing image segmentation based on a U-net model, which is an image segmentation model. At least one of the patch, manhole, road mark, and pothole may be a target to be detected through the image segmentation operation.

[0121] FIG. 16 is a diagram showing an example of a label clustering operation for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0122] The road surface condition determination device performs label clustering by object class in the detected second class area (521). The road surface condition determination device can generate a label list containing different labels (522, 523) by performing label clustering by object class in the second class area (521), which is a segmentation image. For example, the road surface condition determination device can cluster road marks and potholes into different object groups in the second class area (521) and label the clustered road marks and potholes differently. In this way, the road surface condition determination device can cluster adjacent pixels having the same class (e.g., road marks, patches, manholes, and potholes, etc.) into a single cluster and label each cluster with separate label information.

[0123] FIG. 17 is a diagram showing an example of a noise filtering operation for generating second object data in a road surface determination method according to an embodiment of the present invention.

[0124] The road surface condition determination device generates second object data by removing noise from a label list clustered with different labels (522, 523). The second object data may include a label list from which noise has been removed. The road surface condition determination device can remove noise from incorrect labels in the label clustered label list by adding a heuristic rule that includes at least one of size, location, and pothole color. For example, the road surface condition determination device may consider an object labeled as the second object as incorrectly detected noise and filter it out when the object is too small compared to the preset size of the pothole or has a bright color. As shown in FIG. 7, one of the three potholes detected in FIG. 16 is filtered out as noise.

[0125] FIGS. 18 and 19 are drawings illustrating examples of a custom U-net model for an operation to generate second object data in a road surface determination method according to an embodiment of the present invention.

[0126] The road surface condition determination device can detect a second class area containing at least one of a patch, manhole, roadmark, and pothole by performing image segmentation based on a U-net model in which the encoder and decoder are configured in a U-shape on a scaled second area of ​​interest image (512).

[0127] In a road surface condition determination device, image segmentation can be performed based on a custom U-net model (202) of an Edge AI structure in which the number of channels of the U-net model is adjusted from a U-net model (201) of a Server AI structure. Here, the depth of the U-net model is not adjusted, but the width of the model's channels can be adjusted. Through this, the road surface condition determination device can perform image segmentation based on the U-net model to reduce the amount of computation while maintaining the detection performance of second objects such as potholes.

[0128] In this way, a road surface condition determination device according to one embodiment of the present invention can perform image segmentation based on a custom U-net model (532) in which the number of channels of the U-net model (531) is adjusted. Additionally, the road surface condition determination device can perform image segmentation based on a custom U-net model (532) in which the backbone network of the U-net model (531) is replaced. Additionally, the road surface condition determination device can perform image segmentation based on a custom U-net model (532) in which the dataset class of the U-net model (531) is changed and data augmentation is performed.

[0129] FIG. 20 is a configuration diagram of a road surface condition determination device using multiple models according to one embodiment of the present invention.

[0130] As illustrated in FIG. 20, a road surface condition determination device (600) using a plurality of models according to one embodiment of the present invention includes a memory (610) and a processor (620). However, not all of the illustrated components are essential components. The road surface condition determination device (600) may be implemented with more components than illustrated, or with fewer components.

[0131] Below, the specific configuration and operation of each component of the road surface condition determination device (600) of FIG. 20 will be described.

[0132] The memory (610) stores one or more programs related to road surface condition determination operations using multiple models.

[0133] The processor (620) executes one or more programs stored in memory (610).

[0134] The processor (620) detects potential hazardous objects based on an object detection model in an image of the road surface to localize potential hazardous areas, identifies potential hazardous objects in pixel units using a semantic segmentation model in the localized potential hazardous areas, and determines the road surface condition by analyzing the risk level of the identified potential hazardous objects.

[0135] According to the embodiments, the processor (620) can output at least one of object information of an identified potential risk object, a risk level of an analyzed potential risk object, and a determined road surface condition.

[0136] According to embodiments, the processor (620) can detect a first class area by performing object detection based on an object detection model in a road surface image and generate first object data for the detected first class area.

[0137] According to the embodiments, the processor (620) can detect a second class region by performing image segmentation based on a segmentation model in a localized potential risk region, and generate second object data by label clustering the detected second class region.

[0138] According to the embodiments, the processor (620) can track the detected potential risk object by additionally using a tracker after the potential risk object is detected.

[0139] According to the embodiments, the processor (620) can analyze the risk level of a potential risk object using the number of pixels or bounding box area of ​​the identified potential risk object.

[0140] According to the embodiments, the processor (620) can analyze the risk level of a potential risk object into one of a normal state, a warning state, and a bad state according to the risk ratio of [Equation 1].

[0141] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0142] In addition, according to one embodiment of the present invention, the method according to the various embodiments described above may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed online in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0143] Additionally, according to one embodiment of the present invention, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0144] Meanwhile, computer instructions for performing processing operations of the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When computer instructions stored in such a non-transitory computer-readable medium are executed by the processor of a specific device, they cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of a non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0145] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0146] Although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. A method for determining the condition of a road surface performed by a road surface condition determination device, A step of detecting potential hazardous objects based on an object detection model in a road surface image and localizing potential hazardous areas; A step of identifying potential risk objects in pixel units using a Semantic Segmentation Model in the localized potential risk area; and A method for determining road surface conditions using multiple models, comprising the step of determining road surface conditions by analyzing the risk level of the identified potential risk objects.

2. In Paragraph 1, A method for determining road surface conditions using multiple models, further comprising the step of outputting at least one of object information of the identified potential risk object, the risk level of the analyzed potential risk object, and the determined road surface condition.

3. In Paragraph 1, The step of localizing the aforementioned potential risk area is, A method for determining road surface conditions using multiple models, wherein object detection is performed based on an object detection model in the above road surface image to detect a first class region, and first object data for the detected first class region is generated.

4. In Paragraph 1, The step of identifying the above potential risk objects is, A method for determining road surface conditions using multiple models, wherein image segmentation is performed based on a segmentation model in the localized potential risk area to detect a second class area, and label clustering is performed on the detected second class area to generate second object data.

5. In Paragraph 1, A method for determining road surface conditions using multiple models, further comprising the step of tracking the detected potential risk object using an additional tracker after the potential risk object is detected.

6. In Paragraph 1, The step of determining the road surface condition above is, A method for determining road surface conditions using multiple models, which analyzes the risk level of a potential risk object using the number of pixels or bounding box area of ​​the identified potential risk object.

7. In Paragraph 1, The step of determining the road surface condition above is, Here, ratio is the risk ratio, c 픽셀 The number of pixels in the crack, p 영역 ε represents the area of ​​the pothole, w represents the weight of the pothole, and A method for determining road surface conditions using multiple models, which analyzes the risk level of the potential risk object into one of a normal state, a warning state, and a bad state according to the risk ratio of the above mathematical formula.

8. Memory for storing one or more programs; and It includes a processor that executes one or more of the above-mentioned stored programs, and The above processor is, Detecting potential hazardous objects based on an object detection model in road surface images to localize potential hazardous areas, and In the aforementioned localized potential risk area, potential risk objects are identified at the pixel level using a Semantic Segmentation Model, and A road surface condition determination device using multiple models that determines the road surface condition by analyzing the risk level of the identified potential risk object.

9. In Paragraph 8, The above processor is, A road surface condition determination device using multiple models, which outputs at least one of object information of the identified potential risk object, the risk level of the analyzed potential risk object, and the determined road surface condition.

10. In Paragraph 8, The above processor is, A road surface condition determination device using multiple models, which performs object detection based on an object detection model in the above road surface image to detect a first class region and generates first object data for the detected first class region.

11. In Paragraph 8, The above processor is, A road surface condition determination device using multiple models, which performs image segmentation based on a segmentation model in the localized potential risk area to detect a second class area, and generates second object data by label clustering the detected second class area.

12. In Paragraph 8, The above processor is, A road surface condition determination device using multiple models, which tracks the detected potential risk object using an additional tracker after the potential risk object is detected.

13. In Paragraph 8, The above processor is, A road surface condition determination device using multiple models that analyzes the risk level of a potential risk object using the number of pixels or bounding box area of ​​the identified potential risk object.

14. In Paragraph 8, The above processor is, Here, ratio is the risk ratio, c 픽셀 The number of pixels in the crack, p 영역 ε represents the area of ​​the pothole, w represents the weight of the pothole, and A road surface condition determination device using multiple models that analyzes the risk level of the potential risk object into one of a normal state, a warning state, and a bad state according to the risk ratio of the above mathematical formula.

Images