Fire detection device and method using smoke diffusion flow

Abstract

A fire detection device and method using smoke diffusion flow are disclosed. The method by which the fire detection device detects a fire comprises the steps of: collecting a plurality of pieces of training image information; performing first training in which the training image information is applied to a detection model that detects an object, thereby detecting a bounding box containing smoke; calculating sensitivity to the bounding box by using the detection model for which the first training has been completed; generating a training input vector on the basis of the training image information, the coordinates of the bounding box and the sensitivity; and performing second training in which the training input vector is applied to a sensing model obtained by separating a neck and a head in a backbone of the detection model into an object sensing layer and a sensitivity inference layer, thereby detecting a bounding box containing smoke and estimating the sensitivity of the bounding box.

Description

Fire detection device and method utilizing smoke diffusion flow

[0001] The present invention relates to a fire detection device, and more specifically, to a fire detection device and method utilizing a smoke diffusion flow that detects small smoke in real time to support early response to a fire.

[0002] In general, early detection and monitoring are crucial for effective fire management and control. However, conventional fire detection methods often miss the early stages where fire control is possible, resulting in significant damage.

[0003] To complement this, fire analysis systems based on real-time environmental monitoring that apply artificial intelligence and data processing technologies are being used recently.

[0004] However, while these fire analysis systems offer many advantages in terms of ensuring accuracy and supporting real-time response, they require expensive hardware due to the system load caused by processing large amounts of data. Furthermore, they present a problem where real-time response becomes difficult as computational speeds slow down over time due to continuous data collection and analysis.

[0005] The problem that the present invention aims to solve is to provide a fire detection device and method utilizing smoke diffusion flow that analyzes multiple image information to calculate changes in smoke diffusion and supports early response by monitoring a fire in real time based on the calculated results.

[0006] A fire detection method performed by a fire detection device according to the present invention to solve the above problem comprises: a step of collecting a plurality of training image information; a step of performing a first training that detects a bounding box containing smoke by applying the training image information to a detection model that detects objects; a step of calculating sensitivity for the bounding box using the detection model in which the first training is completed; a step of generating a training input vector based on the training image information, the coordinates of the bounding box, and the sensitivity; and a step of performing a second training that detects a bounding box containing smoke and estimates the sensitivity of the bounding box by applying the training input vector to a detection model in which the neck and head of the backbone of the detection model are divided into an object detection layer and a sensitivity inference layer.

[0007] In addition, the collecting step is characterized by performing image processing on the training image information of the daytime environment among the collected plurality of training image information under the conditions of the night environment in order to improve the accuracy of the sensitivity in the night environment.

[0008] In addition, the above-mentioned calculation step is characterized by calculating the sensitivity using the normalized value for the dynamic change of the bounding box, the angle between the changing bounding boxes, and the accuracy for the changing bounding boxes.

[0009] In addition, the above-mentioned calculation step is characterized by calculating the sensitivity by assigning weights according to importance to each of the above-mentioned normalized value, the above-mentioned angle, and the above-mentioned accuracy.

[0010] In addition, the step of performing the second learning described above is characterized by comparing the coordinate values ​​of the bounding box output from the detection model and the sensitivity values ​​with predetermined correct answers, respectively, and iteratively learning while varying the loss function until the detection model reaches an accuracy greater than or equal to a preset level.

[0011] In addition, the step of performing the second learning described above is characterized by performing learning related to the detection of the bounding box using a Cross Entropy loss function, the mean value of CIoU (Complete-IoU), and a Distribution Focal Loss function, and performing learning related to the sensitivity of the bounding box using a Regression loss function.

[0012] In addition, the method is characterized by further including the steps of: preprocessing the collected multiple image information to generate an input vector applicable to the detection model when a plurality of image information for fire detection is collected after the second learning is completed; applying the input vector to the detection model to calculate bounding box-related coordinate values ​​and sensitivity values; and determining whether a fire has occurred using the calculated coordinate values ​​and sensitivity values.

[0013] In addition, the method is characterized by further including, after the above-mentioned judgment step, a step of determining whether smoke is included in the plurality of image information, and if smoke is included in at least one of the plurality of image information, a step of adjusting the resolution of the next image information collected in the order following the image information, and then controlling it to be applied to the detection model.

[0014] In addition, the above-mentioned controlling step is characterized by adjusting the resolution of the next image information so that it increases as the sensitivity value increases.

[0015] A fire detection device according to the present invention includes a learning unit that, when a plurality of learning image information is collected, applies the learning image information to a detection model that detects objects to perform a first learning to detect a bounding box containing smoke, calculates the sensitivity of the bounding box using the detection model that has completed the first learning, generates a learning input vector based on the learning image information, the coordinates of the bounding box, and the sensitivity, and applies the learning input vector to a detection model that divides the neck and head of the backbone of the detection model into an object detection layer and a sensitivity inference layer to perform a second learning to detect a bounding box containing smoke and estimate the sensitivity of the bounding box.

[0016] In addition, the learning unit is characterized by performing image processing on the learning image information of the daytime environment among the collected plurality of learning image information under the conditions of the night environment in order to improve the accuracy of the sensitivity in the night environment.

[0017] In addition, the learning unit is characterized by calculating the sensitivity using the normalized value for the dynamic change of the bounding box, the angle between the changing bounding boxes, and the accuracy for the changing bounding boxes.

[0018] In addition, the learning unit is characterized by comparing the coordinate values ​​and sensitivity values ​​of the bounding box output from the detection model with predetermined correct answers, respectively, and iteratively learning while varying the loss function until the detection model reaches an accuracy greater than or equal to a preset level.

[0019] In addition, the smoke detection unit, which preprocesses the collected multiple image information to generate an input vector applicable to the detection model when multiple image information for fire detection is collected, and applies the input vector to a pre-trained detection model to calculate bounding box-related coordinate values ​​and sensitivity values, and the fire determination unit, which determines whether a fire exists using the calculated coordinate values ​​and sensitivity values, are further included.

[0020] In addition, the fire determination unit determines whether smoke is included in the plurality of image information, and if smoke is included in at least one of the plurality of image information, it controls the next image information collected in the order following the image information to adjust the resolution of each of the next image information, and then applies it to the detection model.

[0021] According to an embodiment of the present invention, considering that the characteristics of smoke diffusion may differ over time for each image information generated from each image capturing device, the image resolution is adjusted based on the individual diffusion sensitivity of the images included in the image information, thereby minimizing the loss of detection accuracy and securing real-time constraints.

[0022] In other words, it can provide fast computation speed and accuracy by maintaining a constant system load, enabling real-time fire detection without the need for high-cost hardware.

[0023] This enables the detection of fires in the early stages and supports rapid fire suppression.

[0024] FIG. 1 is a configuration diagram for explaining a fire detection system according to an embodiment of the present invention.

[0025] FIG. 2 is a block diagram illustrating a fire detection device according to an embodiment of the present invention.

[0026] FIG. 3 is a block diagram for explaining a control unit according to an embodiment of the present invention.

[0027] FIG. 4 is a schematic diagram illustrating a learning process according to an embodiment of the present invention.

[0028] FIG. 5 is a diagram illustrating image information in a daytime environment and image information converted into a nighttime environment through image processing according to an embodiment of the present invention.

[0029] FIG. 6 is a drawing for illustrating a bounding box according to an embodiment of the present invention.

[0030] FIG. 7 is a diagram illustrating the structure of a sensing model according to an embodiment of the present invention.

[0031] FIG. 8 is a drawing illustrating the outermost distance between bounding boxes according to an embodiment of the present invention.

[0032] FIG. 9 is a flowchart illustrating a fire detection method for learning a detection model according to an embodiment of the present invention.

[0033] FIG. 10 is a flowchart illustrating a fire detection method for determining whether a fire has occurred according to an embodiment of the present invention.

[0034] FIG. 11 is a diagram comparing the sensitivity of clouds and smoke based on a fire detection device according to an embodiment of the present invention.

[0035] FIG. 12 is a block diagram for explaining a computing device according to an embodiment of the present invention.

[0036] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0037] In this specification and drawings (hereinafter referred to as the 'this specification'), redundant descriptions of identical components are omitted.

[0038] Furthermore, when a component is described in this specification as being 'connected' or 'connected' to another component, it should be understood that it may be directly connected to or connected to the other component, or that there may be other components in between. On the other hand, when a component is described in this specification as being 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

[0039] Furthermore, the terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention.

[0040] Additionally, in this specification, singular expressions may include plural expressions unless the context clearly indicates otherwise.

[0041] Furthermore, in this specification, terms such as 'comprising' or 'having' are intended merely to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0042] Additionally, in this specification, the term "and / or" includes a combination of the plurality of described items or any of the plurality of described items. In this specification, "A or B" may include "A," "B," or "both A and B."

[0043] In addition, detailed descriptions of known functions and configurations that may obscure the essence of the invention will be omitted in this specification.

[0044]

[0045] FIG. 1 is a configuration diagram for explaining a fire detection system according to an embodiment of the present invention.

[0046] Referring to FIG. 1, the fire detection system (400) analyzes multiple video information to calculate changes in smoke diffusion and supports early response by monitoring the fire in real time based on the calculated results. The fire detection system (400) includes a fire detection device (100), a plurality of video recording devices (200), and a user terminal (300).

[0047] A fire detection device (100) detects a fire at an early stage using multiple image information. Preferably, the fire detection device (100) may be optimized for wildfire disasters, but is not limited thereto. The fire detection device (100) analyzes multiple image information received from multiple image capturing devices (200), and can reduce the load on the system by using minimal resources. To this end, the fire detection device (100) calculates the sensitivity of smoke included in the image information. Sensitivity is a numerical value representing the degree of change of the bounding box that detects smoke, and refers to the direction of the smoke or the degree to which the smoke fades; preferably, it can represent the amount of change in size of the bounding box, the amount of change in position, and the accuracy of smoke detection. For example, if the sensitivity is high, the smoke is faint, increasing the likelihood that it is not a fire, and if the sensitivity is low, the smoke is thick, increasing the likelihood that it is a fire. The fire detection device (100) can optimize system operation by adjusting the resolution of the image information based on the calculated sensitivity. For example, the fire detection device (100) can perform concentrated and selective system operation by increasing the resolution of high-importance image information and decreasing the resolution of low-importance image information. The fire detection device (100) determines whether there is a fire based on the analyzed results and transmits the determined results in real time to the user terminal (300) to support the user in taking quick action regarding the fire.

[0048] A plurality of video recording devices (200) are installed at pre-set locations and generate video information for the installed locations. Preferably, the video recording devices (200) may be implemented as CCTV (Closed-circuit Television) devices, but are not limited thereto. At this time, each video recording device (200a, 200b, 200c, 200n) may be an RGB camera, and depending on the installation environment, it may be a fixed camera with a fixed shooting angle or a rotating camera with a variable shooting angle. Additionally, each video recording device (200a, 200b, 200c, 200n) may generate video information in real time and transmit the generated video information to a fire detection device (100).

[0049] The user terminal (300) is a terminal used by a user (or administrator) and supports the user in monitoring fire detection remotely. The user terminal (300) receives a fire-related judgment result from the fire detection device (100) and outputs the received judgment result. At this time, the user terminal (300) does not simply output whether a fire has occurred, but can also output an event where smoke is detected even if the occurrence of a fire is unclear. Preferably, the user terminal (300) can output video information showing the location where a fire or smoke has occurred from various angles to help the user intuitively recognize the situation at that location.

[0050] Meanwhile, the fire detection system (400) supports communication between the fire detection device (100), a plurality of image capturing devices (200), and a user terminal (300) by establishing a communication network (450). The communication network (450) may be composed of a backbone network and a subscriber network. The backbone network may be composed of one or more integrated networks among an X.25 network, a Frame Relay network, an ATM network, an MPLS (Multi-Protocol Label Switching) network, and a GMPLS (Generalized Multi-Protocol Label Switching) network. The subscriber network may be FTTH (Fiber To The Home), ADSL (Asymmetric Digital Subscriber Line), cable network, Zigbee, Bluetooth, Wireless LAN (IEEE 802.11b, IEEE 802.11a, IEEE 802.11g, IEEE 802.11n), Wireless Hart (ISO / IEC 62591-1), ISA 100.11a (ISO / IEC 62734), CoAP (Constrained Application Protocol), MQTT (Message Queuing Telemetry Transport), WIBro (Wireless Broadband), WiMAX, 3G, HSDPA (High Speed ​​Downlink Packet Access), 4G, 5G, and 6G, etc. In some embodiments, the communication network (450) may be an internet network and a mobile communication network. Additionally, the communication network (450) may include any other widely known or future-developed wireless or wired communication methods.

[0051]

[0052] FIG. 2 is a block diagram illustrating a fire detection device according to an embodiment of the present invention.

[0053] Referring to FIG. 2, the fire detection device (100) includes a communication unit (10), an input unit (20), a control unit (30), and a storage unit (40).

[0054] The communication unit (10) performs communication with a plurality of video recording devices (200) and a user terminal (300). The communication unit (10) receives video information for learning and real-time video information for actual fire detection from the plurality of video recording devices (200). Here, the video information for learning and the real-time video information may be information in the form of continuous images. The communication unit (10) transmits fire-related judgment results to the user terminal (300).

[0055] The input unit (20) can directly receive learning video information from the user. Preferably, the input unit (20) receives learning video information when learning video information is not received through the communication unit (10), but is not limited thereto.

[0056] The control unit (30) performs overall control of the fire detection device (100). The control unit (30) learns a detection model for detecting fire based on video information for training. The detection model is an artificial intelligence model with an additional network that calculates smoke-related sensitivity based on a known object detection model. When an input vector for training related to video information or an input vector for fire detection is input, the detection model outputs the coordinate values ​​of the bounding box and the sensitivity values ​​of the input vector. When the learning of the detection model is completed, the control unit (30) collects video information for detecting actual fire and applies the input vector for the collected video information to the previously trained detection model to calculate the coordinate values ​​of the bounding box and the sensitivity values. The control unit (30) analyzes the smoke state using the calculated bounding box coordinate values ​​and sensitivity values ​​and determines whether there is a fire based on the analyzed results. The control unit (30) transmits the determination result to a user terminal (300). Here, the control unit (30) determines whether smoke is included in each analyzed image information, and if there is image information containing smoke, it adjusts the resolution of the next image information collected in the next order from the image capturing device (200) that generated the image information by reflecting the sensitivity value corresponding to the image information. Through this, the control unit (30) adjusts the resolution of the image information collected in real time, and the resolution may be higher for image information that has high importance for fire detection judgment. Based on the image information with adjusted resolution, the control unit (30) can repeat the fire detection process described above until smoke is not detected in the image information.

[0057] The storage unit (40) stores a program or algorithm for operating the fire detection device (100). The storage unit (40) may store results collected, generated, calculated, and determined for fire detection (e.g., image information, input vector, bounding box coordinate values, bounding box sensitivity values, etc.). The storage unit (40) may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk, etc.

[0058]

[0059] FIG. 3 is a block diagram for explaining a control unit according to an embodiment of the present invention, FIG. 4 is a schematic diagram for explaining a learning process according to an embodiment of the present invention, FIG. 5 is a diagram for explaining image information in a daytime environment and image information converted to a nighttime environment through image processing according to an embodiment of the present invention, FIG. 6 is a diagram for explaining white balance adjustment according to an embodiment of the present invention, FIG. 7 is a diagram for explaining a bounding box according to an embodiment of the present invention, FIG. 8 is a diagram for explaining the structure of a detection model according to an embodiment of the present invention, and FIG. 9 is a diagram for explaining the outermost distance between bounding boxes according to an embodiment of the present invention.

[0060] Referring to FIGS. 1 to 9, the control unit (30) includes a learning unit (31) and may further include a smoke detection unit (32) and a fire determination unit (33).

[0061] The learning unit (31) collects multiple learning image information (Fig. 4 (a)). The learning image information is information collected to train a detection model and may be in the form of an image containing smoke. Additionally, the learning image information may be collected from an image capturing device (200) or directly input and collected by a user.

[0062] The learning unit (31) performs a process of refining collected image information. The learning unit (31) supports obtaining more learning image information by additionally performing image processing on the learning image information in a daytime environment among the collected multiple learning image information under nighttime environment conditions to improve the accuracy of sensitivity in a nighttime environment. That is, the learning unit (31) can convert and generate an image in a daytime environment (Fig. 5 (a)) into an image in a nighttime environment (Fig. 5 (b)), and the generated image in a nighttime environment can be used as learning image information. At this time, the learning unit (31) can perform image processing using at least one algorithm among Inverse Tone Mapping (ITM), Inverse Gamma Correction (IGC), Standard RGB to Camera RGB conversion, Inverse White Balance (IWB), Darkness, Shot and Read Noise, and Quantization Noise.

[0063] Inverse Tone Mapping (ITM) is a technology that converts Standard Dynamic Range (SDR) content into High Dynamic Range (HDR) content. It converts images with low color depth (8-bit) and a screen brightness range of approximately 0 to 100 nits into a wide brightness range (up to 10,000 nits or more) and high color depth (at least 10 bits). The conversion method may be a method of expanding SDR data using fixed values.

[0064] Inverse Gamma Correction (IGC) converts a gamma-corrected image back into its original linear form. Images appear darker than the original when displayed on a monitor, and gamma correction is used to correct this issue. In other words, Inverse Gamma Correction reverses this process, returning the image to its uncorrected state.

[0065] Standard RGB to Camera RGB conversion transforms an image from the standard RGB color space to the camera RGB color space. In other words, standard RGB to camera RGB conversion is an image processing step that extends the color range of standard RGB to the color range of camera RGB.

[0066] Inverse White Balance (IWB) is an image processing technique that reverses white balance. White balance is the process by which a digital camera adjusts the color tone of an image captured under specific lighting conditions to make it appear natural, as seen by the human eye. During this process, the camera balances colors by adjusting the intensity of red, yellow, and blue based on the lighting. For example, blue is made stronger under yellow lighting, and red is made stronger under blue lighting. Therefore, Inverse White Balance is the process of reverting these adjustments to their original state, thereby removing lighting corrections that would otherwise appear unnatural.

[0067] Darkness is an image processing technique that lowers the brightness of an image. Darkness reduces the overall brightness of the image, making it darker.

[0068] Shot and read noise applies noise that can occur in actual low-light environments by combining shot noise and read noise. In this case, shot and read noise applies both types of noise using fixed values. Shot noise refers to noise generated in an image due to the characteristic that the amount of incident photons in a specific area of ​​the sensor is random and not constant over time. Read noise is a form of electronic noise generated in digital image sensors, occurring during the process in which the image sensor converts signals into digital values.

[0069] Quantization noise is an error that occurs because an analog signal is rounded or truncated to the nearest digital value. In this case, quantization noise is applied using a fixed value.

[0070] When the refining process is completed, the learning unit (31) applies the refined learning image information to a detection model (M) that detects objects and performs a first learning process to detect a bounding box (B) containing smoke (Fig. 4(b)). The detection model (M) is an artificial intelligence model to which a known object recognition algorithm is applied, and preferably, it may be a YOLO model, but is not limited thereto. At this time, the learning unit (31) can output the coordinate values ​​of the bounding box (B) for the smoke portion included in the image of the learning image information using the detection model (M).

[0071] The learning unit (31) calculates sensitivity to bounding boxes using the detection model (M) that has completed the first learning ((c) of FIG. 4). The learning unit (31) calculates sensitivity using the normalized value for the dynamic change of the bounding box, the angle between the changing bounding boxes, and the accuracy for the changing bounding boxes. At this time, the learning unit (31) can calculate sensitivity by assigning weights according to importance to each of the normalized value, the angle between, and the accuracy.

[0072] More specifically, the learning unit (31) [as in Mathematical Formula 1] is the upper-left corner (x) of the bounding box (B) detected by the detection model (M). min , y min ) and bottom right (x max , y max Calculate the area of ​​the bounding box by multiplying the difference between ).

[0073]

[0074] Here represents the area of ​​the bounding box for the i-th object in the t-frame image of the image information, and x max refers to the x-coordinate of the bottom-right, and x min represents the x-coordinate of the top-left corner, and y max represents the y-coordinate of the bottom-right, and y min represents the y-coordinate of the top-left corner.

[0075] The learning unit (31) calculates a normalized value by normalizing the difference in area of ​​the bounding box (B) to a value between 0 and 1 as in [Mathematical Formula 2].

[0076]

[0077] Here represents the normalized value that is dynamically changed from the bounding box for the i-th object in the t-1 frame image of the image information to the bounding box for the i-th object in the t-frame image of the image information.

[0078] The learning unit (31) also calculates the center coordinates of the bounding box (B) as in [Equation 3].

[0079]

[0080] Here represents the center coordinates of the bounding box for the i-th object in the t-frame image of the image information, and x c represents the x-coordinate among the center coordinates, and y c represents the y-coordinate among the center coordinates.

[0081] In addition, the learning unit (31) calculates the angle between bounding boxes (B) that change in a time series as [Equation 4].

[0082]

[0083] Here represents the angle between the center coordinates of the bounding box for the i-th object in the t-1 frame of the image information and the center coordinates of the bounding box for the i-th object in the t-frame of the image information, and represents the y-coordinate of the center of the bounding box for the i-th object in the t-frame image, and represents the y-coordinate of the center of the bounding box for the i-th object in the t-1 frame image, and represents the x-coordinate of the center of the bounding box for the i-th object in the t-frame image, and represents the x-coordinate of the center of the bounding box for the i-th object in the t-1 frame image.

[0084] The learning unit (31) can calculate the accuracy of the bounding box (B) that changes based on the smoke detection status of the frame image. That is, the learning unit (31) defines the case where smoke is detected in a frame image with smoke as TP, the case where there is no detected result for a frame image without smoke as TN, the case where smoke is detected even though there is no smoke in the frame image as FP, and the case where smoke is detected even though there is smoke in the frame image as FN. The learning unit (31) calculates the accuracy of the bounding box (B) based on the number of cases defined as [Equation 5].

[0085]

[0086] Here, Precision refers to the ratio of actual correct answers among those classified as correct by the detection model, Recall refers to the ratio of actual correct answers predicted by the detection model, and F1_Score refers to the accuracy of the bounding box.

[0087] At this time, the learning unit (31) can calculate sensitivity after assigning weights using hyperparameters α, β, and γ. α, β, and γ are values ​​set by the user, and as long as the sum of the three values ​​is 1 (α+β+γ=1), they can be set to any value. That is, the values ​​of α, β, and γ can be varied according to the importance of the item corresponding to each variable.

[0088] The learning unit (31) can calculate the bounding box-related sensitivity as in [Equation 6] and calculate the average sensitivity value within a single frame image as in [Equation 7].

[0089]

[0090]

[0091] Here represents the sensitivity value of the bounding box for the i-th object in the t-frame image of the image information, and represents the average sensitivity value of bounding boxes within the t-frame image of the image information, and N represents the number of objects (number of bounding boxes) within the t-frame image.

[0092] The learning unit (31) generates a learning input vector based on the learning image information, the coordinates of the bounding box, and the sensitivity of the bounding box (Fig. 4 (d)). The learning input vector may be an input value that is input to the detection model (DM).

[0093] The learning unit (31) applies the generated learning input vector to the detection model (DM) to perform a second learning process for smoke detection and sensitivity estimation (Fig. 4(e)). The detection model (DM) has a structure in which the neck and head are divided into an object detection layer (NH1) and a sensitivity inference layer (NH2) in the backbone (BB) of the first learned detection model (M). That is, the detection model (DM) can have a structure in which the sensitivity inference layer (NH2) is added in a redundant configuration in the neck and head of the first learned detection model (M). Through this, the detection model (DM) can output not only the coordinate values ​​of the bounding box but also the sensitivity values ​​of the bounding box as output values. At this time, the object detection layer (NH1) uses the Cross Entropy loss function, the mean value of CIoU (Complete-IoU), and the Distribution Focal Loss function, and the sensitivity inference layer (NH2) uses the Regression loss function.

[0094] More specifically, the cross-entropy loss function is expressed as [Equation 8], the average value of CIoU is expressed as [Equation 9], and the distribution focus loss function can be expressed as [Equation 10].

[0095]

[0096] Here, BCE represents the binary cross-entropy loss function, and i represents the number of objects within a frame; for example, if there are 5 objects in a frame, i corresponds to 0, 1, 2, 3, 4, and represents the center y-coordinate of the bounding box drawn by the person on the i-th object as the correct answer, and represents the center y-coordinate of the bounding box for the i-th object predicted by the detection model.

[0097]

[0098] Here, avg(CIoU) means the average value of CIoU, and represents the CIoU of the bounding box for the i-th object, but, .... In this case, DIoU refers to distance-IoU loss, but, And, is, IoU stands for Intersection over Union and is a value between 0 and 1 that represents the degree of agreement between the predicted bounding box and the actual facts (Ground Truth). It is expressed as. B is the center coordinate value of the predicted bounding box, It is expressed as, where x represents the center x-coordinate of the bounding box corresponding to the object predicted by the detection model, y represents the center y-coordinate of the bounding box corresponding to the object predicted by the detection model, w represents the width of the bounding box corresponding to the object predicted by the detection model, and h represents the height of the bounding box corresponding to the object predicted by the detection model. B gt is the center coordinate value of the correct bounding box, Expressed as, x gt represents the x-coordinate of the center of the bounding box corresponding to the object drawn by the person as the correct answer, and y gt represents the y-coordinate of the center of the bounding box corresponding to the object drawn by the person as the correct answer, and w gt represents the width of the bounding box corresponding to the object drawn by the person as the correct answer, and h gt represents the height of the bounding box corresponding to the object drawn by the person as the correct answer. Euc is the Euclidean distance, and c represents the outermost distance between two bounding boxes (B1, B2) (Fig. 8).

[0099]

[0100] Here, DFL refers to the distributed focus loss function.

[0101] The learning unit (31) can calculate the loss value of the object detection layer (NH1) using the cross-entropy loss function, the average value of CIoU, and the distribution focus loss function as [Equation 11].

[0102]

[0103] Here, Detection loss refers to the loss value of the object detection layer (NH1).

[0104] In addition, the regression loss function is expressed as [Equation 12].

[0105]

[0106] Here, Sensitivity loss refers to the loss value of the sensitivity inference layer (NH2), and L1 loss It refers to the loss function used when the result of applying the absolute value to the difference between the correct y-value and the y-value predicted by the detection model is greater than or equal to 1, and it calculates the Mean Absolute Error (MAE) between the predicted and actual values ​​to exhibit robustness against outliers. It is expressed as, and L2 loss It refers to the loss function used when the result of applying the absolute value of the difference between the correct y-value and the y-value predicted by the detection model is less than 1, and it calculates the Mean Squared Error (MSE) between the predicted and actual values ​​to exhibit characteristics sensitive to outliers. It is expressed as.

[0107] The learning unit (31) can finally calculate the loss value of the detection model (DM) using [Equation 13].

[0108]

[0109] The learning unit (31) performs a second learning process to optimize the loss value of the detection model (DM). That is, the learning unit (31) compares the coordinate values ​​and sensitivity values ​​of the bounding box (B) output from the detection model (DM) with a predetermined correct answer value, respectively, and can perform iterative learning by varying the loss function until the detection model (DM) reaches an accuracy level higher than a preset level. Here, the accuracy can be determined by the user, but is not limited thereto.

[0110] The smoke detection unit (32) performs actual fire detection using a detection model (DM) that has completed a second learning by the learning unit (31). To this end, when multiple image information for fire detection is collected from multiple image capturing devices (200), the smoke detection unit (32) preprocesses the collected multiple image information to generate an input vector applicable to the detection model (DM). The smoke detection unit (32) applies the generated input vector to the previously learned detection model to calculate bounding box-related coordinate values ​​and sensitivity values.

[0111] The fire detection unit (33) determines whether a fire has occurred using the calculated coordinate values ​​and sensitivity values ​​related to the bounding box. At this time, the fire detection unit (33) does not determine whether a fire has occurred based on the coordinate values ​​and sensitivity values ​​corresponding to a single image, but rather determines whether a fire has occurred based on the coordinate values ​​and sensitivity values ​​corresponding to multiple angle images obtained at the same time and the coordinate values ​​and sensitivity values ​​corresponding to multiple angle images obtained at the next time. Through this, the fire detection unit (33) can improve the accuracy of fire detection by detecting smoke that is not visible in the image taken on one side in the image taken on the other side and determining whether a fire has occurred. Additionally, the fire detection unit (33) does not determine that a fire has occurred merely because smoke has been generated, but can minimize fire detection errors through the amount of change in the bounding box that detected the smoke. Preferably, the fire detection unit (33) can detect fire in multiple grades, and the fire grades can be divided into, for example, initial, caution, warning, danger, etc. The fire judgment unit (33) controls the transmission of the determined result to the user terminal (300) so that the result can be checked on the user terminal (300).

[0112] Additionally, the fire determination unit (33) determines whether smoke is included in a plurality of image information that has been determined to be a fire, and if smoke is included in at least one of the plurality of image information, it adjusts the resolution of the next image information by reflecting the sensitivity value output from the detection model (DM) to the next image information collected in the order following the image information. That is, the fire determination unit (33) adjusts the resolution of the next image information so that it is higher as the sensitivity value is higher, and lowers the resolution of the next image information as the sensitivity value is lower. Through this, the fire determination unit (33) can support concentrated and selective system operation by adjusting the resolution of image information collected in chronological order in real time to increase the resolution of image information of high importance and decrease the resolution of image information of low importance.

[0113]

[0114] FIG. 10 is a flowchart illustrating a fire detection method for learning a detection model according to an embodiment of the present invention.

[0115] Referring to FIGS. 1 and FIGS. 10, the fire detection method can maximize the precision of the result value by performing learning of a dual-structure detection model.

[0116] In step S110, the fire detection device (100) collects multiple training image information. The fire detection device (100) can collect training image information to train a detection model. At this time, the fire detection device (100) can collect the information from multiple image capturing devices (200) or by receiving the information directly from a user. At this time, the fire detection device (100) can perform image processing on the training image information of the daytime environment among the collected multiple training image information under the conditions of a night environment in order to improve the accuracy of the detection model in a night environment.

[0117] In step S120, the fire detection device (100) performs a first learning. The fire detection device (100) applies the learning image information to a detection model that detects objects to perform a first learning that detects a bounding box containing smoke. The detection model is an artificial intelligence model to which a known object recognition algorithm is applied, and preferably may be a YOLO model, but is not limited thereto.

[0118] In step S130, the fire detection device (100) calculates the sensitivity to the bounding box using the detection model that has completed the first learning. The fire detection device (100) can calculate the sensitivity using the normalized value for the dynamic change of the bounding box, the angle between the changing bounding box, and the accuracy for the changing bounding box. At this time, the fire detection device (100) can calculate the sensitivity by assigning weights to each of the normalized value, the angle between, and the accuracy according to their importance.

[0119] In step S140, the fire detection device (100) generates a learning input vector. The fire detection device (100) generates a learning input vector in a format applicable to a detection model based on learning image information, bounding box coordinates, and sensitivity. Here, the detection model is an artificial intelligence model with an added network that further calculates smoke-related sensitivity based on a known object detection model, and has a structure in which the neck and head are divided into an object detection layer and a sensitivity inference layer based on the backbone of the detection model that has completed the first training.

[0120] In step S150, the fire detection device (100) performs a second learning. The fire detection device (100) applies a learning input vector to a detection model to perform a second learning that detects a bounding box containing smoke and estimates the sensitivity of the bounding box. The fire detection device (100) compares the coordinate values ​​and sensitivity values ​​of the bounding box output from the detection model with a predetermined correct answer value, respectively, and performs iterative learning by varying the loss function until the detection model reaches an accuracy level greater than or equal to a preset level.

[0121]

[0122] FIG. 11 is a flowchart illustrating a fire detection method for determining whether a fire has occurred according to an embodiment of the present invention.

[0123] Referring to FIGS. 1 and 11, the fire detection method minimizes loss of detection accuracy and secures real-time constraints by adjusting the image resolution based on the individual diffusion sensitivity of the images included in the image information, taking into account that the characteristics of smoke diffusion may differ over time for each image information generated from each image capturing device. In other words, the fire detection method can provide fast computation speed and accuracy by maintaining a constant system load so that real-time fire detection can be performed without high-cost hardware. Through this, the fire detection method can detect a fire at an early stage and support rapid fire suppression.

[0124] In step S210, the fire detection device (100) collects multiple image information for fire detection. The fire detection device (100) can collect multiple image information in real time from multiple image capturing devices (200). Here, the multiple image information may be images captured from multiple angles.

[0125] In step S220, the fire detection device (100) preprocesses the collected multiple image information. The fire detection device (100) preprocesses the collected multiple image information to generate an input vector applicable to the detection model. At this time, the fire detection device (100) may perform image processing to improve the clarity of the collected multiple image information and remove pixel errors, but is not limited thereto.

[0126] In step S230, the fire detection device (100) applies the generated input vector to a pre-trained detection model to produce a smoke-related output. At this time, the fire detection device (100) can calculate the coordinate values ​​and sensitivity values ​​related to the bounding box containing smoke using the second detection model that has completed training.

[0127] In step S240, the fire detection device (100) determines whether there is a fire based on the smoke state. The fire detection device (100) determines whether there is a fire using the calculated boundary box related coordinate values ​​and sensitivity values. At this time, the fire detection device (100) can divide the fire state into multiple grades. For example, the fire detection device (100) can divide the fire grades into initial, caution, warning, danger, etc.

[0128] In step S250, the fire detection device (100) transmits the judgment result to the user terminal (300). The fire detection device (100) can transmit the judgment result to the user terminal (300) in real time. Preferably, the fire detection device (100) can further transmit video information that serves as the basis for determining that it is a fire, thereby helping the user intuitively recognize the fire condition through said information.

[0129] In step S260, the fire detection device (100) determines whether smoke is included in a plurality of image information. If smoke is included in at least one of the plurality of image information, the fire detection device (100) performs step S270, and if it is not included, terminates the system operation.

[0130] In step S270, the fire detection device (100) adjusts the resolution of the next image information by reflecting a sensitivity value. The fire detection device (100) branches between steps S210 and S220 and adjusts the resolution of the next image information by reflecting the corresponding sensitivity value to the next image information collected in the next order of the image information containing multiple smoke (the latest image information re-collected in step S210 in chronological order). At this time, the fire detection device (100) can support intensive analysis of high-importance image information by adjusting the resolution of the next image information so that the higher the sensitivity value, the higher the resolution of the next image information.

[0131]

[0132] FIG. 12 is a diagram comparing the sensitivity of clouds and smoke based on a fire detection device according to an embodiment of the present invention.

[0133] Referring to FIG. 12, the fire detection device (100) can distinguish between clouds and smoke based on sensitivity. Here, FIG. 12 (a) is a diagram showing cloud-related sensitivity before normalization, and FIG. 12 (b) is a diagram showing smoke-related sensitivity before normalization. Inliers refer to the reference range of the threshold value, and outliers refer to the actual measured sensitivity.

[0134] Looking at Fig. 12 (a), it can be seen that most of the values ​​are concentrated in the clouds, and looking at Fig. 12 (b), the values ​​are spread out in the smoke.

[0135] That is, the fire detection device (100) can detect fire by distinguishing between clouds and smoke using these characteristics.

[0136]

[0137] As described above, the fire detection device of the present invention allows the system to better understand the situation of object behavior, such as the direction of smoke or fire spread, by considering changes in angle or direction regarding movement. Through this, the fire detection device can adjust image scaling more precisely, thereby preventing the loss of important information and maintaining or improving detection accuracy.

[0138] In addition, through a nuanced understanding of diffusion sensitivity, the fire detection device can make more informed decisions about when and to what extent the system should reduce image size. That is, by identifying images requiring higher resolution based on sensitivity, the system can allocate resources more efficiently, allowing it to pay more attention to computations for high-sensitivity images while using fewer resources for computations for low-sensitivity images.

[0139]

[0140] FIG. 13 is a block diagram for explaining a computing device according to an embodiment of the present invention.

[0141] Referring to FIG. 13, the computing device (TN100) may be a device described in the present specification (e.g., a fire detection device, a plurality of image capturing devices, a user terminal, etc.).

[0142] The computing device (TN100) may include at least one processor (TN110), a transceiver (TN120), and a memory (TN130). Additionally, the computing device (TN100) may further include a storage device (TN140), an input interface device (TN150), an output interface device (TN160), etc. The components included in the computing device (TN100) may be connected by a bus (TN170) to communicate with each other.

[0143] The processor (TN110) can execute a program command stored in at least one of the memory (TN130) and the storage device (TN140). The processor (TN110) may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. The processor (TN110) may be configured to implement the procedures, functions, and methods described in relation to embodiments of the present invention. The processor (TN110) can control each component of the computing device (TN100).

[0144] Each of the memory (TN130) and the storage device (TN140) can store various information related to the operation of the processor (TN110). Each of the memory (TN130) and the storage device (TN140) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (TN130) may be composed of at least one of read-only memory (ROM) and random access memory (RAM).

[0145] The transmitting and receiving device (TN120) can transmit or receive wired or wireless signals. The transmitting and receiving device (TN120) can be connected to a network to perform communication.

[0146]

[0147] Meanwhile, embodiments of the present invention are not limited to being implemented only through the apparatus and / or methods described so far, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which such program is recorded, and such implementation can be easily achieved by a person skilled in the art to which the present invention belongs based on the description of the embodiments described above.

[0148]

[0149] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by a person skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

1. A fire detection method performed by a fire detection device, A step of collecting multiple learning image information; A step of performing a first learning step of detecting a bounding box containing smoke by applying the above-mentioned training image information to a detection model that detects objects; A step of calculating sensitivity for the bounding box using the detection model that has completed the first learning above; A step of generating a training input vector based on the training image information, the coordinates of the bounding box, and the sensitivity; and A step of performing a second learning process by applying the above-mentioned learning input vector to a detection model in which the neck and head are divided into an object detection layer and a sensitivity inference layer on the backbone of the detection model to detect bounding boxes containing smoke and estimate the sensitivity of said bounding boxes; A fire detection method including 2. In Paragraph 1, The above-mentioned collecting step is, A fire detection method characterized by performing image processing on training image information in a daytime environment among the collected plurality of training image information under nighttime environment conditions to improve the accuracy of the sensitivity in a nighttime environment.

3. In Paragraph 1, The above-mentioned calculation step is, A fire detection method characterized by calculating the sensitivity using a normalized value for the dynamic change of the bounding box, the angle between the changing bounding box, and the accuracy for the changing bounding box.

4. In Paragraph 3, The above-mentioned calculation step is, A fire detection method characterized by calculating the sensitivity by assigning weights according to importance to each of the normalized value, the angle between, and the accuracy.

5. In Paragraph 1, The step of performing the above second learning is, A fire detection method characterized by comparing the coordinate values ​​and sensitivity values ​​of a bounding box output from the above detection model with a predetermined correct answer value, and iteratively learning while varying the loss function until the detection model reaches an accuracy greater than a preset level.

6. In Paragraph 5, The step of performing the above second learning is, A fire detection method characterized by performing learning related to detection of the bounding box using a cross entropy loss function, the mean value of CIoU (Complete-IoU), and a distribution focal loss function, and performing learning related to sensitivity of the bounding box using a regression loss function.

7. In Paragraph 1, After the above second learning is completed, A step of preprocessing the collected multiple image information when multiple image information for fire detection is collected to generate an input vector applicable to the detection model; A step of applying the above input vector to the above detection model to calculate bounding box-related coordinate values ​​and sensitivity values; and A step of determining whether there is a fire using the coordinate values ​​and sensitivity values ​​calculated above; A fire detection method characterized by further including 8. In Paragraph 7, After the above-mentioned judgment step, A step of determining whether the plurality of image information above contains smoke; If smoke is included in at least one of the plurality of image information, a step of reflecting the sensitivity value to the next image information collected in the order following the image information, adjusting the resolution of each of the next image information, and then controlling it to be applied to the detection model; A fire detection method characterized by further including 9. In Paragraph 8, The above-mentioned controlling step is, A fire detection method characterized by adjusting the resolution of the following image information so that the higher the sensitivity value, the higher the resolution of the following image information.

10. A learning unit that, when multiple training image information is collected, applies the training image information to a detection model that detects objects to perform a first learning to detect bounding boxes containing smoke, calculates the sensitivity for the bounding boxes using the detection model that has completed the first learning, generates a training input vector based on the training image information, the coordinates of the bounding boxes, and the sensitivity, and applies the training input vector to a detection model in which the neck and head are divided into an object detection layer and a sensitivity inference layer on the backbone of the detection model to perform a second learning to detect bounding boxes containing smoke and estimate the sensitivity of the bounding boxes; A fire detection device including 11. In Paragraph 10, The above learning unit is, A fire detection device characterized by performing image processing on training image information in a daytime environment among the collected plurality of training image information under nighttime environment conditions to improve the accuracy of the sensitivity in a nighttime environment.

12. In Paragraph 10, The above learning unit is, A fire detection device characterized by calculating the sensitivity using a normalized value for the dynamic change of the bounding box, the angle between the changing bounding boxes, and the accuracy for the changing bounding boxes.

13. In Paragraph 10, The above learning unit is, A fire detection device characterized by comparing the coordinate values ​​and sensitivity values ​​of a bounding box output from the above detection model with a predetermined correct answer value, respectively, and iteratively learning while varying the loss function until the detection model reaches an accuracy greater than a preset level.

14. In Paragraph 10, A smoke detection unit that, when multiple image information for fire detection is collected, preprocesses the collected multiple image information to generate an input vector applicable to the detection model, and applies the input vector to a pre-trained detection model to calculate bounding box-related coordinate values ​​and sensitivity values; and A fire determination unit that determines whether a fire has occurred using the above-determined coordinate values ​​and sensitivity values; A fire detection device characterized by further including 15. In Paragraph 14, The above fire determination unit is, A fire detection device characterized by determining whether smoke is included in the plurality of image information, and if smoke is included in at least one of the plurality of image information, reflecting the sensitivity value to the next image information collected in the order following the image information to adjust the resolution of the next image information, and then controlling it to be applied to the detection model.

Images