Detection method and system for leaked gas based on multi-data fusion, and electronic device

Abstract

A detection method and system for a leaked gas based on multi-data fusion, and an electronic device, relating to the technical field of gas detection. The method comprises: acquiring a plurality of compressed images obtained by respectively compressing a plurality of first infrared images; on the basis of the plurality of compressed images, predicting a position of a potential leaked gas as a candidate position; extracting, respectively from a plurality of second infrared images, sub-images at the candidate position as infrared sub-images; and on the basis of all of the infrared sub-images, predicting a first classification result for the candidate position, wherein the first classification result is used for indicating the existence of a leaked gas at the candidate position or the absence of a leaked gas at the candidate position. By means of embodiments of the present application, rapid and accurate detection of a leaked gas can be achieved.

Description

Methods, systems, and electronic devices for detecting leaked gas based on multi-data mixing

[0001] This application claims priority to Chinese Patent Application No. 202411649772.8, filed on November 18, 2024, entitled "Method, Apparatus and Electronic Equipment for Detecting Leaked Gas Based on Multi-Data Mixing", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of gas detection technology, and in particular to methods, systems and electronic devices for detecting leaked gases based on multi-data mixing. Background Technology

[0003] Gas leaks can cause serious safety accidents or environmental pollution in industrial production, transportation, and daily life, threatening people's lives and causing property damage. Gas leak detection allows for the timely identification and handling of potential gas leak sources, effectively preventing these safety accidents.

[0004] In related technologies, gas leak detection methods mainly fall into two categories: The first category is based on traditional image processing and machine learning. Its process involves: firstly, using an infrared camera to capture the current scene and obtain video image data; then, enhancing the video image to reduce noise and enhance gas cloud features; next, performing differential calculations based on adjacent video frames to obtain the moving target region; finally, classifying the moving target region, determining the classification result as a gas cloud, and triggering an alarm. However, this type of method performs poorly in complex environments such as complex scenes, low temperature differences, and weak ambient light, resulting in low detection accuracy. The second category of gas leak detection schemes introduces a powerful deep neural network to simultaneously process the infrared image of the current scene and... The visible light image processing flow is as follows: First, the infrared image is fed into a leak gas target detection model to obtain preliminary suspected locations of leak gas. Then, the visible light image is fed into a scene understanding model, and based on the output scene region segmentation map, the preliminary suspected locations of leak gas are filtered, removing suspected gas locations within specific areas. For example, removing suspected gas locations in the sky can reduce false alarms caused by clouds, thus obtaining highly suspected locations of leak gas. Next, the multi-frame temporal images from the infrared channel are fed into a gas sequence classification model to further filter the highly suspected locations of leak gas, obtaining the final leak gas alarm location. This type of method directly uses data acquired by the camera or directly processes image data obtained after data compression. Using data acquired by the camera can optimize model performance, improve the ability to handle complex scenes, and accurately obtain the location of leak gas alarms in the scene. However, using data results in large storage requirements and slow transmission speeds, which will increase product costs and extend the model iteration cycle, thus reducing detection efficiency. Directly processing image data obtained after data compression can reduce product transmission costs, but it will lead to lower detection accuracy.

[0005] It is evident that the relevant technologies for detecting leaked gases suffer from low accuracy or low efficiency. How to detect leaked gases quickly and accurately has become a pressing technical problem to be solved in this field. Summary of the Invention

[0006] The purpose of this application is to provide a method, system, and electronic device for detecting leaked gas based on multi-data mixing, so as to achieve rapid and accurate detection of leaked gas. The specific technical solution is as follows:

[0007] In a first aspect, embodiments of this application provide a leak gas detection method based on multi-data mixing, the method comprising:

[0008] Multiple compressed images are obtained by compressing multiple first infrared images respectively, wherein the first infrared images are uncompressed infrared images obtained by capturing the target scene;

[0009] The locations of potential gas leaks are predicted based on the multiple compressed images and used as candidate locations.

[0010] Sub-images at the candidate locations are extracted from multiple second infrared images and used as infrared sub-images, wherein the second infrared images are uncompressed infrared images obtained by capturing the target scene, and the capture time span of the multiple first infrared images is a proper subset of the capture time span of the multiple second infrared images;

[0011] A first classification result is obtained from all the infrared sub-images to predict the candidate location. The first classification result is used to indicate that there is a gas leak at the candidate location, or that there is no gas leak at the candidate location.

[0012] In one possible implementation, the step of predicting the first classification result of the candidate location based on all the infrared sub-images includes:

[0013] Extract image features from at least one of the infrared sub-images as infrared spatial features; and extract sequence features from the image sequence formed by sorting all the infrared sub-images in chronological order of their capture time as infrared sequence features;

[0014] Based on the infrared spatial features and the infrared sequence features, the first classification result of the candidate location is predicted.

[0015] In one possible implementation, extracting image features from at least one of the infrared sub-images as infrared spatial features includes:

[0016] Gas scores for all the infrared sub-images are predicted respectively, wherein the gas scores are positively correlated with the number of features of leaked gas contained in the image;

[0017] The image whose gas score meets the preset screening criteria is selected from all the infrared sub-images and used as the target infrared sub-image;

[0018] The image features of the target infrared sub-image are extracted as infrared spatial features.

[0019] In one possible implementation, the method further includes:

[0020] Sub-images at the candidate locations are extracted from multiple visible light images and used as visible light sub-images. The visible light images are uncompressed visible light images obtained by capturing the target scene, and the time span of capturing the multiple second infrared images is the same as the time span of capturing the multiple visible light images.

[0021] A second classification result is obtained from all the visible photon images to predict the candidate location. The second classification result is used to indicate that there is a moving object at the candidate location, or that there is no moving object at the candidate location.

[0022] If the second classification result indicates that there is no moving object, and the first classification result indicates that there is a leaking gas, then it is determined that there is a leaking gas at the candidate location.

[0023] In one possible implementation, the step of predicting the second classification result of the candidate location based on all the visible photon images includes:

[0024] Image features of the target visible photon image are extracted as visible light spatial features; and sequence features of the image sequence formed by sorting all the visible photon images in chronological order of their capture time are extracted as visible light sequence features.

[0025] Based on the visible light spatial features and the visible light sequence features, a second classification result for the candidate location is predicted;

[0026] The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

[0027] In one possible implementation, the method further includes:

[0028] Acquire uncompressed infrared images of the target scene captured within the first time period preceding the time to be detected, and use them as the first infrared image;

[0029] An uncompressed infrared image of the target scene captured within a second time period preceding the time to be detected is obtained as a second infrared image, wherein the second time period is longer than the first time period.

[0030] Secondly, embodiments of this application provide a leak gas detection device based on multi-data mixing, the device comprising:

[0031] The first acquisition module is used to acquire multiple compressed images obtained by compressing multiple first infrared images respectively, wherein the first infrared image is an uncompressed infrared image obtained by shooting the target scene;

[0032] The location prediction module is used to predict the location of potential leaking gas based on the multiple compressed images, and to select the candidate location.

[0033] The first extraction module is used to extract sub-images at the candidate positions from multiple second infrared images as infrared sub-images, wherein the second infrared images are uncompressed infrared images obtained by capturing the target scene, and the capture time span of the multiple first infrared images is a proper subset of the capture time span of the multiple second infrared images.

[0034] The first classification module is used to predict a first classification result of the candidate location based on all the infrared sub-images. The first classification result is used to indicate that there is gas leakage at the candidate location, or that there is no gas leakage at the candidate location.

[0035] In one possible implementation, the first classification module includes:

[0036] The first classification submodule is used to extract image features of at least one of the infrared sub-images as infrared spatial features; and to extract sequence features of the image sequence formed by sorting all the infrared sub-images in chronological order of shooting time as infrared sequence features;

[0037] The second classification submodule is used to predict the first classification result of the candidate position based on the infrared spatial features and the infrared sequence features.

[0038] In one possible implementation, the first classification submodule includes:

[0039] The first classification unit is used to predict the gas score of all the infrared sub-images respectively, wherein the gas score is positively correlated with the number of features of leaked gas contained in the image;

[0040] The second classification unit is used to determine, from all the infrared sub-images, the image whose gas score meets the preset screening conditions, as the target infrared sub-image;

[0041] The third classification unit is used to extract image features from the infrared sub-image of the target as infrared spatial features.

[0042] In one possible implementation, the device further includes:

[0043] The second extraction module is used to extract sub-images at the candidate positions from multiple visible light images as visible light sub-images, wherein the visible light images are uncompressed visible light images obtained by capturing the target scene, and the time span of capturing the multiple second infrared images is the same as the time span of capturing the multiple visible light images.

[0044] The second classification module is used to predict a second classification result of the candidate position based on all the visible photon images. The second classification result is used to indicate that there is a moving object at the candidate position, or that there is no moving object at the candidate position.

[0045] The result determination module is used to determine that there is a gas leak at the candidate location if the second classification result indicates that there is no moving object and the first classification result indicates that there is a gas leak.

[0046] In one possible implementation, the second classification module includes:

[0047] The fourth classification unit is used to extract image features from the target visible photon image as visible light spatial features; and to extract sequence features from the image sequence formed by sorting all the visible photon images in chronological order of shooting time as visible light sequence features;

[0048] The fifth classification unit is used to predict the second classification result of the candidate position based on the visible light spatial features and the visible light sequence features;

[0049] The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

[0050] In one possible implementation, the device further includes:

[0051] The second acquisition module is used to acquire uncompressed infrared images of the target scene captured within a first time period preceding the start of the detection time, as the first infrared image;

[0052] The third acquisition module is used to acquire an uncompressed infrared image of the target scene taken within a second time period preceding the time to be detected, as a second infrared image, wherein the second time period is longer than the first time period.

[0053] Thirdly, embodiments of this application provide a leak gas detection system based on multi-data mixing, the system including a first image acquisition device and a processor;

[0054] The first image acquisition device is used to capture a target scene to obtain multiple first infrared images and multiple second infrared images, wherein the time span of capturing the multiple first infrared images is a proper subset of the time span of capturing the multiple second infrared images, and the first infrared images and the second infrared images are uncompressed infrared images;

[0055] The processor is configured to predict the location of a potential gas leak based on the plurality of first infrared images and the second infrared image, as a candidate location, and predict a first classification result for the candidate location, wherein the first classification result is used to indicate that there is a gas leak at the candidate location, or that there is no gas leak at the candidate location.

[0056] In one possible implementation, the processor predicts the location of a potential leaking gas based on the plurality of first infrared images and the second infrared image, including:

[0057] The plurality of first infrared images are compressed to obtain a plurality of compressed images;

[0058] The location of the potential leaking gas is predicted based on the multiple compressed images;

[0059] Sub-images at the candidate locations are extracted from multiple second infrared images and used as infrared sub-images;

[0060] The first classification result of the candidate location is obtained based on predictions from all the infrared sub-images.

[0061] In one possible implementation, the step of predicting the first classification result of the candidate location based on all the infrared sub-images includes:

[0062] Extract image features from at least one of the infrared sub-images as infrared spatial features; and extract sequence features from the image sequence formed by sorting all the infrared sub-images in chronological order of their capture time as infrared sequence features;

[0063] Based on the infrared spatial features and the infrared sequence features, the first classification result of the candidate location is predicted.

[0064] In one possible implementation, extracting image features from at least one of the infrared sub-images as infrared spatial features includes:

[0065] Gas scores for all the infrared sub-images are predicted respectively, wherein the gas scores are positively correlated with the number of features of leaked gas contained in the image;

[0066] The image whose gas score meets the preset screening criteria is selected from all the infrared sub-images and used as the target infrared sub-image;

[0067] The image features of the target infrared sub-image are extracted as infrared spatial features.

[0068] In one possible implementation, the system further includes a second image acquisition device, which is used to capture multiple visible light images of the target scene, wherein the time span for capturing the multiple second infrared images is the same as the time span for capturing the multiple visible light images.

[0069] The processor is further configured to extract sub-images at the candidate locations from multiple visible light images, respectively, as visible light sub-images; and to predict a second classification result of the candidate location based on all the visible light sub-images, wherein the second classification result is used to indicate that there is a moving object at the candidate location, or that there is no moving object at the candidate location;

[0070] If the second classification result indicates that there is no moving object, and the first classification result indicates that there is a gas leak, then it is determined that there is a gas leak at the candidate location.

[0071] In one possible implementation, the processor predicts a second classification result for the candidate location based on all the visible photon images, including:

[0072] Image features of the target visible photon image are extracted as visible light spatial features; and sequence features of the image sequence formed by sorting all the visible photon images in chronological order of their capture time are extracted as visible light sequence features.

[0073] Based on the visible light spatial features and the visible light sequence features, a second classification result for the candidate location is predicted;

[0074] The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

[0075] In one possible implementation, the first infrared image is obtained by the first image acquisition device capturing the target scene within a first time period preceding the time to be detected, and the second infrared image is obtained by the first image acquisition device capturing the target scene within a second time period preceding the time to be detected.

[0076] Fourthly, embodiments of this application provide an electronic device, including:

[0077] Memory, used to store computer programs;

[0078] When the processor executes the program stored in the memory, it implements any of the above-described methods for detecting leaked gas based on multi-data mixing.

[0079] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for detecting leaked gas based on multi-data mixing.

[0080] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the above-described methods for detecting leaked gas based on multi-data mixing.

[0081] Beneficial effects of the embodiments in this application:

[0082] The leak gas detection method based on multi-data mixing provided in this application first predicts candidate locations of potential leak gas based on compressed infrared images of the target scene. Then, it extracts infrared sub-images of each candidate location from each infrared image. Finally, it predicts whether leak gas exists at the candidate location based on each infrared sub-image. With this method, the compressed infrared image has a small data volume, resulting in relatively fast processing speed. Determining candidate locations based on the compressed infrared image can quickly obtain each candidate location. Furthermore, since the infrared image contains more comprehensive information than the compressed infrared image, after obtaining candidate locations of potential leak gas based on the compressed infrared image, by extracting sub-images of each candidate location from the infrared image, it is possible to accurately determine whether gas leaks exist at each candidate location, thereby achieving rapid and accurate detection of leak gas.

[0083] Furthermore, the step of predicting candidate locations of potential leaking gas based on compressed infrared images of the target scene can be implemented using a pre-trained model. During the training of this model, the input is also the compressed infrared image. Since image compression is relatively inexpensive, if the current scene is complex, the model can be iterated in a short time, improving its scene adaptability. Moreover, the step of predicting the first classification result of the candidate locations based on all infrared sub-images can also be implemented using a pre-trained model. During the training of this model, the input is also the infrared sub-image. Since each infrared sub-image is a sub-image of a candidate location within the infrared image, and candidate locations are typically small, the model's processing task is relatively simple. Therefore, the model does not require frequent iterations and updates, thus achieving fast and accurate leaking gas detection while keeping costs low or minimally increased.

[0084] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0085] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0086] Figure 1 is a flowchart of the first type of leak gas detection method based on multi-data mixing provided in the embodiments of this application;

[0087] Figure 2 is a flowchart of the leak gas sequence detection model provided in the embodiment of this application;

[0088] Figure 3 is a second flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application;

[0089] Figure 4 is a third flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application;

[0090] Figure 5 is a flowchart of the leakage gas sequence classification model provided in the embodiments of this application;

[0091] Figure 6 is a fourth flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application;

[0092] Figure 7 is a fifth flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application;

[0093] Figure 8 is a flowchart of the visible light target analysis model provided in the embodiments of this application;

[0094] Figure 9 is a flowchart illustrating the leak gas detection method based on multi-data mixing provided in an embodiment of this application.

[0095] Figure 10 is a schematic diagram of the structure of the leak gas detection device based on multi-data mixing provided in the embodiment of this application;

[0096] Figure 11 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0098] First, the technical terms used in the embodiments of this application will be explained:

[0099] VOCs (Volatile Organic Compounds): Volatile organic compounds, generally classified into several categories such as non-methane hydrocarbons, oxygen-containing organic compounds, halogenated hydrocarbons, nitrogen-containing organic compounds, and sulfur-containing organic compounds.

[0100] Infrared camera: Captures and images light waves of a specific wavelength in the environment, generating a single-channel image.

[0101] Dual-light camera: The camera has two different mechanisms that can image the visible light band and the infrared light band respectively.

[0102] RAW data: In this embodiment of the application, data refers to the data obtained by the infrared camera mechanism capturing ambient light, and the single pixel value is usually represented by a 16-bit width.

[0103] Long-term exposure to VOCs in the air can cause significant harm to human health and environmental pollution. Natural gas, as a clean fossil fuel, plays an important role in daily life, its main component being gaseous low-molecular-weight hydrocarbons. However, due to its high flammability, leaks during use or transportation can easily cause fires or even explosions, posing a significant threat to life and property. Therefore, to prevent fires and explosions and protect the environment, VOCs and natural gas leak detection is typically conducted in industrial processes from production to transportation.

[0104] Since both VOCs and natural gas are invisible in the visible light band but have a significant imaging effect in the mid-infrared band, many leak gas detection algorithms exist in related technologies. Their main processing flow is roughly as follows: First, an infrared camera is used to detect the leak scene and obtain infrared image data; then, relevant data preprocessing methods are used to process the data to reduce noise and enhance gas characteristics; next, based on the data with enhanced gas characteristics, appropriate machine learning or deep learning methods are used to obtain a leak gas location warning; finally, based on the visible light camera imaging results, the leak gas warning is filtered to obtain the final leak gas location alarm.

[0105] The above solutions can achieve intelligent monitoring of gas leaks in various scenarios to a certain extent, but they also have some problems, specifically: 1. Most solutions use traditional manual feature-based methods with low complexity, resulting in limited overall monitoring effectiveness and difficulty in applying them to complex scenarios; 2. A few solutions use cutting-edge deep learning methods to monitor gas leaks. These solutions train deep learning models using data from multiple scenarios, improving their ability to handle complex scenarios. However, training deep learning models typically requires covering a large amount of data from various processing scenarios. Some solutions directly collect 16-bit wide camera RAW data from various scenarios, which optimizes model performance, but RAW data collection suffers from large storage requirements and slow transmission speeds. This significantly increases product costs and prolongs the model iteration cycle, making it difficult to respond promptly to short-term needs at the project site. Some solutions compress RAW data into 8-bit wide video data at the camera end before transmission to achieve large-scale data collection and model training. However, compressing 16-bit wide RAW data into 8-bit wide video data results in significant information loss, blurring the characteristics of rarefied gases and reducing the model's alarm capability for such gases.

[0106] Furthermore, infrared camera imaging is inherently unstable; even if a certain area in the image remains unchanged, the pixel values ​​of adjacent video frames may differ significantly. In real-world processing scenarios, numerous non-cloud moving objects exist, such as vehicles, people, shaking pipes, swaying leaves, drifting clouds, and camera shake caused by wind. These factors can all lead to the differential algorithm generating a large number of suspicious targets, hindering accurate detection of cloud targets.

[0107] To improve the accuracy and efficiency of leak gas detection, a first aspect of this application provides a leak gas detection method based on multi-data mixing, which can be applied to electronic devices. In specific applications, the electronic device can be a server or a terminal device, both of which are within the scope of protection of this application.

[0108] Referring to Figure 1, which is a first flowchart of a leak gas detection method based on multi-data mixing provided in an embodiment of this application, the method includes the following steps:

[0109] Step S1: Obtain multiple compressed images obtained by compressing multiple first infrared images respectively;

[0110] The first infrared image is an uncompressed infrared image obtained by capturing the target scene;

[0111] Step S2: The location of the potential leaking gas is predicted based on multiple compressed images and used as candidate locations;

[0112] Step S3: Extract sub-images at candidate locations from multiple second infrared images, and use them as infrared sub-images;

[0113] The second infrared image is an uncompressed infrared image obtained by shooting the target scene, and the shooting time span of the multiple first infrared images is a proper subset of the shooting time span of the multiple second infrared images; both the first infrared image and the second infrared image are raw image data obtained by the image acquisition device used to shoot the target scene, i.e., RAW data.

[0114] Step S4: Based on the prediction of all infrared sub-images, the first classification result of the candidate location is obtained. The first classification result is used to indicate whether there is gas leakage at the candidate location or whether there is no gas leakage at the candidate location.

[0115] The method of this application embodiment has a small data volume and relatively fast processing speed due to the small data volume of the compressed infrared image. The candidate positions can be quickly obtained by determining the candidate positions based on the compressed infrared image. Furthermore, since the information contained in the infrared image is more comprehensive than that contained in the compressed infrared image, after obtaining the candidate positions of potential leaking gas based on the compressed infrared image, the sub-images of each candidate position in the infrared image can be extracted. Based on each sub-image, it can be accurately determined whether there is a gas leak at each candidate position, thereby realizing the rapid and accurate detection of leaking gas.

[0116] Furthermore, the step of predicting candidate locations of potential leaking gas based on compressed infrared images of the target scene can be implemented using a pre-trained model. During the training of this model, the input is also the compressed infrared image. Since image compression is relatively inexpensive, if the current scene is complex, the model can be iterated in a short time, improving its scene adaptability. Similarly, the step of predicting the first classification result of the candidate locations based on all infrared sub-images can also be implemented using a pre-trained model. During the training of this model, the input is also the infrared sub-image. Since each infrared sub-image is a sub-image of a candidate location within the infrared image, and candidate locations are typically small, the model's processing task is relatively simple. Therefore, the model does not require frequent iterations, thus achieving fast and accurate leaking gas detection while keeping costs low or minimal. Detailed explanations of the models for predicting candidate locations of potential leaking gas and the models for predicting the first classification result of candidate locations are provided below and will not be repeated here.

[0117] The following will explain steps S1 to S4:

[0118] In step S1, the first infrared image can be an uncompressed infrared image of the target scene captured by an infrared camera, or an uncompressed infrared image of the target scene captured by a dual-light camera.

[0119] The first infrared image captured by an infrared camera module typically has a bit width of 16 bits, meaning that each pixel has a depth of 16 bits and can represent 65,536 different gray levels.

[0120] Compressing the first infrared image to obtain multiple compressed images means deframing the first infrared image and compressing it into images with a bit width of less than 16, such as deframing and compressing the first infrared image into multiple compressed images with a bit width of 8 bits.

[0121] In step S2, the location of the potential leaking gas is predicted based on multiple compressed images. This can be achieved by using a deep model trained by machine learning or by using a traditional model trained by other machine learning algorithms.

[0122] The depth model used to determine the location of potential gas leaks, trained using machine learning, is referred to below as the leak gas sequence detection model. In the leak gas sequence detection model, compressed infrared images (i.e., compressed images) are used as input for prediction, and the output is the location of the potential gas leak.

[0123] Figure 2 shows a flowchart of the leak gas sequence detection model provided in this application embodiment. The leak gas sequence detection model mainly includes a multi-frame image feature extraction module, a target coarse generation module, and a target selection module.

[0124] After compressing the infrared images obtained from the infrared channel to obtain compressed images, the multi-frame image feature extraction module receives multiple compressed images at different times as input. By capturing the full spatial information and temporal information from each compressed image, it extracts the gas feature information from each compressed image (hereinafter referred to as the image sequence) to obtain the feature extraction result. Then, the feature extraction result is input into the target coarse generation module, which outputs the approximate location of the foreground target in the image (corresponding to the foreground target box in the figure). The leak gas sequence detection model includes multiple target selection modules, each consisting of multiple convolutional layers and fully connected layers. Since the target coarse generation module only outputs the approximate location of the foreground target, multiple target selection modules continuously refine the approximate location of the foreground target to obtain the accurate location of the potential leaking gas. For example, the foreground target box is input into target selection module 1, which outputs gas selection box 1. Then, gas selection box 1 is input into target selection module 2, which outputs gas selection box 2. Then, gas selection box 2 is input into target selection module 3, which outputs gas selection box 3. Finally, gas selection box 3 is used as the output of the leak gas sequence detection model to obtain the location of the potential leaking gas, i.e., the suspected location of the leaking gas.

[0125] The multi-frame image feature extraction module can use a ConvNext network, or a deep convolutional network such as ResNet or DenseNet; these are all acceptable, and this application does not limit the specific implementation. The target coarse generation module can use an RPN network, or other networks; this application also does not limit the specific implementation.

[0126] As shown in Figure 2, in the process of training the leak gas sequence detection model, each module of the model can be trained using compressed images of samples with ground truth labels for gas locations. The model is then optimized based on the calculated loss function to obtain the trained leak gas sequence detection model. The loss function includes, but is not limited to, cross-entropy loss and mean squared error loss.

[0127] The leak gas sequence detection model can simultaneously capture both spatial and temporal information in the target scene by using a sequence input method. Furthermore, it makes predictions based on compressed images, requiring less data and enabling rapid determination of the location of each potential leaking gas.

[0128] However, the processing capacity of the leak gas sequence detection model is limited, and the temporal information contained in the input image sequence is also limited, usually 1 to 2 seconds in length. This can lead to false alarms for some targets that are similar to the imaging characteristics of leak gas, such as false alarms caused by changes in shadows. In the subsequent steps S3 and S4, when verifying the location of each potential leak gas, image frames with a longer time span can be selected.

[0129] In step S3, the second infrared image is also obtained by an infrared camera or a dual-light camera capturing the target scene without compression. The time span of capturing multiple first infrared images is shorter than the time span of capturing multiple second infrared images, and the multiple first infrared images are a subset of the multiple second infrared images. That is, the time span of capturing multiple first infrared images is a proper subset of the time span of capturing multiple second infrared images. For example, if 10 consecutive image frames are captured of the target scene within 5 seconds of the current time, and these frames are numbered from image frame 1 to image frame 10, then the first infrared image is any at least two consecutively numbered image frames from image frames 1 to image frame 10, such as image frames 1-5, or image frames 2-4.

[0130] Based on the above, to facilitate the acquisition of the first and second infrared images, in one possible implementation, after capturing infrared images of the target scene, each infrared image is stored. Thus, during leak gas detection, the uncompressed infrared images of the target scene captured within a first time period preceding the detection time can be directly retrieved from the stored infrared images as the first infrared image, and the uncompressed infrared images of the target scene captured within a second time period preceding the detection time can be retrieved as the second infrared image, where the second time period is longer than the first time period. Since the second time period is longer than the first time period, the first infrared image can be considered a subset of the second infrared image. This ensures that the first classification result for all candidate locations can be predicted, improving the accuracy of leak gas detection.

[0131] Extracting a sub-image at a candidate location from multiple second infrared images refers to extracting the image corresponding to the candidate location from the second infrared images based on the candidate location. This extraction can be performed using image processing software or by using a depth model trained by machine learning for image extraction. This application does not limit the scope of this embodiment.

[0132] Then, in step S4, the classification result for each candidate location is predicted. This prediction of the classification result based on each infrared sub-image can be achieved either by using a deep model trained through machine learning to determine sequence classification, or by using a traditional model trained through other machine learning algorithms.

[0133] To improve the accuracy of classification results, during the process of predicting the classification results of candidate locations based on all infrared sub-images, spatial and temporal features of the infrared sub-images can be extracted, and predictions can be made based on these spatial and temporal features.

[0134] Based on this, Figure 3 shows a second flowchart of the leak gas detection method based on multi-data mixing provided in this application embodiment. The above step S4 specifically includes:

[0135] Step S41: Extract image features from at least one infrared sub-image as infrared spatial features; and extract sequence features from the image sequence formed by sorting all infrared sub-images in chronological order of shooting time as infrared sequence features;

[0136] Among them, the infrared sequence features reflect the temporal characteristics of all infrared sub-images.

[0137] Step S42: Based on infrared spatial features and infrared sequence features, predict the first classification result of the candidate location.

[0138] The method of this application is used to predict the classification results of candidate positions by extracting spatial and temporal features of infrared sub-images. Since infrared sub-images and spatial and temporal features can more comprehensively reflect the changes in candidate positions, the accuracy of classification results is improved.

[0139] It is understandable that, since gas leakage is dynamic, the number of features of the leaking gas contained in the infrared sub-images at different times will also vary. To ensure that false detections do not occur, in one possible implementation, before spatial feature extraction, the number of image features in each infrared sub-image can be determined first, and each infrared sub-image can be scored according to the number of image features. The higher the score, the more image features the infrared sub-image contains. Then, the infrared sub-image with the highest score is selected for spatial feature extraction.

[0140] Based on this, in one possible implementation, as shown in Figure 4, which is a third flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application, step S41 may specifically include:

[0141] Step S411: Predict the gas scores for all infrared sub-images respectively;

[0142] The gas score is positively correlated with the number of leaked gas features in the image; the more leaked gas features an image contains, the higher its gas score.

[0143] Step S412: Determine the image whose gas score meets the preset screening conditions from all infrared sub-images and use it as the target infrared sub-image;

[0144] The preset screening criteria can refer to a score higher than a preset threshold or the highest score. The preset threshold is determined based on practical experience, and this embodiment does not limit its application.

[0145] Step S413: Extract the image features of the target infrared sub-image as infrared spatial features.

[0146] By using the method of this application embodiment, the target infrared sub-image for spatial feature extraction is determined based on the gas score of each infrared sub-image. This ensures that the extracted spatial features can comprehensively reflect all gas features in the target scene, avoiding missed detection of the location of leaked gas.

[0147] In cases where both infrared image segmentation and candidate location classification prediction are implemented using deep models trained by machine learning, to reduce hardware costs and simplify the operation process, one possible implementation is to fuse the deep model used for image segmentation with the deep model used for classification prediction to obtain a leak gas sequence classification model. In this model, candidate locations and a second infrared image are used as input for prediction, and the classification result for the candidate locations is output.

[0148] Figure 5 shows a flowchart of the leakage gas sequence classification model provided in this application embodiment. The leakage gas sequence classification model includes a data local extraction module, a multi-frame data feature extraction module, and a category classification module. The candidate positions (i.e., suspected leakage gas positions) obtained in step S2 above are input into the data local extraction module. The data local extraction module extracts the image at the candidate position in the infrared image corresponding to the compressed image where the candidate position is located, based on the candidate position, to obtain local multi-frame sequence data, i.e., multiple infrared sub-images.

[0149] Then, the multi-frame data feature extraction module is used to extract features from the local multi-frame data to obtain multi-dimensional features of the candidate positions. Finally, the extracted multi-dimensional features of the candidate positions are input into the category classification module to obtain the classification results.

[0150] The multi-frame data feature extraction module performs feature extraction on local multi-frame data as follows: First, the local multi-frame data is preprocessed, including but not limited to pixel value normalization, compression and amplification, and image increment, to obtain the processed enhanced data; then, the spatial feature extraction submodule and the temporal feature extraction submodule are used to extract features from the enhanced data from both temporal and spatial dimensions to obtain spatial features and temporal features; then, the temporal features and spatial features are input into the feature fusion submodule for fusion to obtain multi-dimensional features of the candidate positions.

[0151] When extracting spatial features, the spatial feature extraction submodule first searches within each infrared image frame sequence to identify keyframes (i.e., infrared image keyframe filtering), and then extracts features from the keyframes to obtain the spatial features of candidate locations. The spatial feature extraction sub-model can be obtained by fusing the Inception_V2 classification network and the DarkNet18 network, or by fusing other classification networks and feature extraction networks; this embodiment does not limit this. The temporal feature extraction network can be obtained by fusing the ResNet18 network and a pseudo-3D network, or by other networks used for temporal feature extraction; this embodiment does not limit this.

[0152] As shown in Figure 5, during the training of the leak gas sequence classification model, after obtaining the sequence classification results, the cross-entropy loss can be used to optimize the leak gas sequence classification model to obtain a well-trained leak gas sequence classification model.

[0153] It is understandable that since gas is invisible in a visible light camera, but other objects have obvious features in a visible light camera, after determining the candidate location according to the above step S2, it is possible to further determine whether a gas leak has occurred at the candidate location based on the changes in the candidate location in both the infrared image and the visible light image.

[0154] Based on this, referring to Figure 6, which is a fourth flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application, the above method further includes:

[0155] Step S5: Extract sub-images at candidate locations from multiple visible light images, and use them as visible light sub-images;

[0156] Among them, the visible light image is an uncompressed visible light image obtained by shooting the target scene, and the shooting time span of the multiple second infrared images is the same as the shooting time span of the multiple visible light images.

[0157] Step S6: Based on the prediction of all visible photon images, the second classification result of the candidate location is obtained. The second classification result is used to indicate whether there is a moving object at the candidate location or whether there is no moving object at the candidate location.

[0158] Step S7: If the second classification result indicates that there is no moving object and the first classification result indicates that there is leaking gas, then it is determined that there is leaking gas at the candidate location.

[0159] By combining visible light images with infrared images to predict the classification results of candidate locations using the embodiments of this application, the false detection problem caused by moving objects or moving light and shadows can be reduced, thereby improving the accuracy of leaked gas detection.

[0160] Similarly to step S4 above, in the process of predicting candidate locations based on the classification results of all visible photon images, predictions can also be made based on the spatial and temporal features of the visible photon images.

[0161] Based on this, in one possible implementation, as shown in Figure 7, which is a fifth flowchart of the leak gas detection method based on multi-data mixing provided in the embodiments of this application, step S6 further includes:

[0162] Step S61: Extract the image features of the target visible photon image as visible light spatial features; and extract the sequence features of the image sequence formed by sorting all visible photon images in chronological order of shooting time as visible light sequence features;

[0163] Because visible light images contain many object attributes, when determining the target visible photon image, the keyframes cannot be determined based on the number of image features. Instead, the visible photon images are arranged in chronological order, and the visible photon image furthest from the beginning and end is taken as the target visible photon image.

[0164] Step S62: Based on the spatial features and sequence features of visible light, predict the second classification result of the candidate position;

[0165] By selecting the embodiments of this application, the classification results of candidate locations are predicted by extracting spatial and temporal features from visible photon images. Since visible photon images and spatial and temporal features can more comprehensively reflect the motion of other objects besides gas, the accuracy of the classification results can be further improved.

[0166] With steps S5 through S7 implemented using a deep model trained via machine learning, a deep model for image extraction and classification prediction can be trained, hereinafter referred to as the visible light target analysis model. Unlike the leaked gas sequence classification model described above, the visible light target analysis model is trained based on visible light images, and its input consists of candidate locations and visible light images, with the output being the classification result of the candidate locations.

[0167] Figure 8 shows a flowchart of the visible light target analysis model provided in this embodiment. The model structure of the visible light target analysis model is the same as that of the leaked gas sequence classification model described above, including a data local extraction module, a multi-frame data feature extraction module, and a category classification module. However, the data local extraction module of the visible light target analysis model is used to extract visible light images based on the candidate positions obtained in step S2 above, and the multi-frame data feature extraction module is used to extract features from the visible light images.

[0168] Specifically, as shown in Figure 8, the candidate positions and visible light images are first input into the local data extraction module. The local data extraction module extracts the image at the candidate position in the visible light image corresponding to the compressed image where the candidate position is located, based on the candidate position, to obtain local multi-frame sequence data, i.e., multiple visible photon images.

[0169] Then, the multi-frame data feature extraction module is used to extract features from the local multi-frame data to obtain multi-dimensional features of the candidate positions. Finally, the extracted multi-dimensional features of the candidate positions are input into the category classification module to obtain the classification results.

[0170] The multi-frame data feature extraction module performs feature extraction on local multi-frame data as follows: First, the visible light data is preprocessed, including but not limited to pixel value normalization, compression and amplification, and image increment, to obtain the processed enhanced data; then, the spatial feature extraction submodule and the temporal feature extraction submodule are used to extract features from the enhanced data from both temporal and spatial dimensions to obtain spatial features and temporal features; then, the temporal features and spatial features are input into the feature fusion submodule for fusion to obtain multi-dimensional features of the candidate location.

[0171] When extracting spatial features, the spatial feature extraction submodule first selects the image frames furthest from the beginning and end of each infrared image frame sequence as keyframes, such as the middle frames (i.e., visible light image keyframe selection); then, it extracts features from the keyframes to obtain the spatial features of candidate locations. The spatial feature extraction submodel can be a DarkNet18 network or other feature extraction networks, and this application embodiment does not limit this; the temporal feature extraction network can be obtained by fusing a ResNet18 network and a pseudo-3D network, or other networks used for temporal feature extraction, and this application embodiment does not limit this.

[0172] As shown in Figure 8, during the training of the visible light target analysis model, after obtaining the sequence classification results, the cross-entropy loss can be used to optimize the visible light target analysis model to obtain a well-trained visible light target analysis model.

[0173] To facilitate timely handling of leaked gas by relevant personnel, an alarm can be triggered based on the classification results obtained above. It is understood that, to achieve automated alarms, in one possible implementation, a result processing and analysis model can be pre-trained. This model can integrate the outputs of the aforementioned models to trigger an alarm at the location of the leaking gas.

[0174] For infrared cameras or dual-light cameras without visible light analysis, the result processing and analysis model directly generates an alarm based on the classification results output by the leak gas sequence classification model; for dual-light cameras with visible light analysis enabled, the result processing and analysis model generates an alarm based on the output results of the visible light target analysis model and the classification results output by the leak gas sequence classification model.

[0175] To more clearly illustrate the leak gas detection method based on multi-data mixing according to the embodiments of this application, the following description is provided in conjunction with specific embodiments. Figure 9 shows a flowchart example of the leak gas detection method based on multi-data mixing provided in the embodiments of this application.

[0176] Taking a dual-light camera as an example, after capturing the scene data, the visible light channel data (i.e., visible light image) and the infrared channel data (i.e., infrared image) are buffered for multiple frames. The infrared image is then compressed, and the compressed image frames are input into an 8-bit wide leak gas sequence detection model to output candidate locations. These candidate locations, along with the infrared image, are then input into a 16-bit wide leak gas sequence classification model to obtain the first classification result, i.e., the leak gas candidate location. The candidate location, along with the visible light image, is then input into a 16-bit wide visible light target analysis model for prediction to obtain the second classification result. Finally, based on the first and second classification results, the results are analyzed, and an alarm is triggered for the candidate location where a gas leak has occurred (i.e., the leak gas alarm location).

[0177] In one possible implementation, the above-mentioned leak gas sequence detection model, leak gas sequence classification model, and visible light target analysis model can be integrated into a single leak gas detection model. In this embodiment, the leak gas detection model is pre-trained according to the following method:

[0178] Multiple compressed sample images are obtained by compressing multiple first infrared sample images respectively, wherein the first infrared sample images are uncompressed infrared images obtained by shooting a preset scene;

[0179] The location of potential leaking gas is predicted based on multiple sample compressed images and used as candidate sample locations;

[0180] Sub-images at the candidate sample locations are extracted from multiple second infrared sample images and used as sample infrared sub-images. The second infrared sample images are uncompressed infrared images with gas location truth labels obtained by capturing the preset scene, and the capture time span of the multiple first infrared sample images is a proper subset of the capture time span of the multiple second infrared sample images.

[0181] A first classification result is obtained from the infrared sub-images of all the samples to predict the candidate locations of the samples. The first classification result is used to indicate that there is gas leakage at the candidate location, or that there is no gas leakage at the candidate location.

[0182] The model parameters of the leak gas detection model are adjusted based on the loss between the candidate sample location and the gas location, as well as the loss between the first classification result and the actual classification result. The process then returns to the above steps of predicting the potential leak gas location based on multiple sample compressed images as candidate sample locations, until the loss between the first classification result and the actual classification result reaches the preset convergence condition.

[0183] Corresponding to the aforementioned leak gas detection method based on multi-data mixing, a second aspect of this application also provides a leak gas detection device based on multi-data mixing. Referring to FIG10, FIG10 is a schematic diagram of the structure of the leak gas detection device based on multi-data mixing provided in this application embodiment, including:

[0184] The first acquisition module 1001 is used to acquire multiple compressed images obtained by compressing multiple first infrared images respectively, wherein the first infrared image is an uncompressed infrared image obtained by shooting the target scene;

[0185] Location prediction module 1002 is used to predict the location of potential leaking gas based on the multiple compressed images, as candidate locations;

[0186] The first extraction module 1003 is used to extract sub-images at the candidate positions from multiple second infrared images as infrared sub-images, wherein the second infrared images are uncompressed infrared images obtained by capturing the target scene, and the capture time span of the multiple first infrared images is a proper subset of the capture time span of the multiple second infrared images.

[0187] The first classification module 1004 is used to predict a first classification result of the candidate location based on all the infrared sub-images. The first classification result is used to indicate that there is a gas leak at the candidate location, or that there is no gas leak at the candidate location.

[0188] Applying the above embodiments, since the compressed infrared image has a small data volume and relatively fast processing speed, the candidate positions can be quickly obtained by determining the candidate positions based on the compressed infrared image. Furthermore, since the infrared image contains more comprehensive information than the compressed infrared image, after obtaining the candidate positions of potential leaking gas based on the compressed infrared image, by extracting the sub-images of each candidate position in the infrared image, it is possible to accurately determine whether there is a gas leak at each candidate position, thereby achieving rapid and accurate detection of leaking gas.

[0189] Furthermore, the step of predicting candidate locations of potential leaking gas based on compressed infrared images of the target scene can be implemented using a pre-trained model. During the training of this model, the input is also the compressed infrared image. Since image compression is relatively inexpensive, if the current scene is complex, the model can be iterated in a short time, improving its scene adaptability. Moreover, the step of predicting the first classification result of the candidate locations based on all infrared sub-images can also be implemented using a pre-trained model. During the training of this model, the input is also the infrared sub-image. Since each infrared sub-image is a sub-image of a candidate location within the infrared image, and candidate locations are typically small, the model's processing task is relatively simple. Therefore, the model does not require frequent iterations and updates, thus achieving fast and accurate leaking gas detection while keeping costs low or minimally increased.

[0190] In one possible implementation, the first classification module 1004 includes:

[0191] The first classification submodule is used to extract image features of at least one of the infrared sub-images as infrared spatial features; and to extract sequence features of the image sequence formed by sorting all the infrared sub-images in chronological order of shooting time as infrared sequence features;

[0192] The second classification submodule is used to predict the first classification result of the candidate position based on the infrared spatial features and the infrared sequence features.

[0193] In one possible implementation, the first classification submodule includes:

[0194] The first classification unit is used to predict the gas score of all the infrared sub-images respectively, wherein the gas score is positively correlated with the number of features of leaked gas contained in the image;

[0195] The second classification unit is used to determine, from all the infrared sub-images, the image whose gas score meets the preset screening conditions, as the target infrared sub-image;

[0196] The third classification unit is used to extract image features from the infrared sub-image of the target as infrared spatial features.

[0197] In one possible implementation, the device further includes:

[0198] The second extraction module is used to extract sub-images at the candidate positions from multiple visible light images as visible light sub-images, wherein the visible light images are uncompressed visible light images obtained by capturing the target scene, and the time span of capturing the multiple second infrared images is the same as the time span of capturing the multiple visible light images.

[0199] The second classification module is used to predict a second classification result of the candidate position based on all the visible photon images. The second classification result is used to indicate that there is a moving object at the candidate position, or that there is no moving object at the candidate position.

[0200] The result determination module is used to determine that there is a gas leak at the candidate location if the second classification result indicates that there is no moving object and the first classification result indicates that there is a gas leak.

[0201] In one possible implementation, the second classification module includes:

[0202] The fourth classification unit is used to extract image features from the target visible photon image as visible light spatial features; and to extract sequence features from the image sequence formed by sorting all the visible photon images in chronological order of shooting time as visible light sequence features;

[0203] The fifth classification unit is used to predict the second classification result of the candidate position based on the visible light spatial features and the visible light sequence features;

[0204] The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

[0205] In one possible implementation, the device further includes:

[0206] The second acquisition module is used to acquire uncompressed infrared images of the target scene captured within a first time period preceding the start of the detection time, as the first infrared image;

[0207] The third acquisition module is used to acquire an uncompressed infrared image of the target scene taken within a second time period preceding the time to be detected, as a second infrared image, wherein the second time period is longer than the first time period.

[0208] Corresponding to the leak gas detection method based on multi-data mixing provided in the first aspect above, the third aspect of the present application also provides a leak gas detection system based on multi-data mixing, the system including a first image acquisition device and a processor;

[0209] The first image acquisition device is used to capture a target scene to obtain multiple first infrared images and multiple second infrared images, wherein the time span of capturing the multiple first infrared images is a proper subset of the time span of capturing the multiple second infrared images, and the first infrared images and the second infrared images are uncompressed infrared images;

[0210] The processor is configured to predict the location of a potential gas leak based on the plurality of first infrared images and the second infrared image, as a candidate location, and predict a first classification result for the candidate location, wherein the first classification result is used to indicate that there is a gas leak at the candidate location, or that there is no gas leak at the candidate location.

[0211] Applying the above embodiments, since the compressed infrared image has a small data volume and relatively fast processing speed, the candidate positions can be quickly obtained by determining the candidate positions based on the compressed infrared image. Furthermore, since the infrared image contains more comprehensive information than the compressed infrared image, after obtaining the candidate positions of potential leaking gas based on the compressed infrared image, by extracting the sub-images of each candidate position in the infrared image, it is possible to accurately determine whether there is a gas leak at each candidate position, thereby achieving rapid and accurate detection of leaking gas.

[0212] Furthermore, the step of predicting candidate locations of potential leaking gas based on compressed infrared images of the target scene can be implemented using a pre-trained model. During the training of this model, the input is also the compressed infrared image. Since image compression is relatively inexpensive, if the current scene is complex, the model can be iterated in a short time, improving its scene adaptability. Moreover, the step of predicting the first classification result of the candidate locations based on all infrared sub-images can also be implemented using a pre-trained model. During the training of this model, the input is also the infrared sub-image. Since each infrared sub-image is a sub-image of a candidate location within the infrared image, and candidate locations are typically small, the model's processing task is relatively simple. Therefore, the model does not require frequent iterations and updates, thus achieving fast and accurate leaking gas detection while keeping costs low or minimally increased.

[0213] In one possible implementation, the processor predicts the location of a potential leaking gas based on the plurality of first infrared images and the second infrared image, including:

[0214] The plurality of first infrared images are compressed to obtain a plurality of compressed images;

[0215] The location of the potential leaking gas is predicted based on the multiple compressed images;

[0216] Sub-images at the candidate locations are extracted from multiple second infrared images and used as infrared sub-images;

[0217] The first classification result of the candidate location is obtained based on predictions from all the infrared sub-images.

[0218] In one possible implementation, the step of predicting the first classification result of the candidate location based on all the infrared sub-images includes:

[0219] Extract image features from at least one of the infrared sub-images as infrared spatial features; and extract sequence features from the image sequence formed by sorting all the infrared sub-images in chronological order of their capture time as infrared sequence features;

[0220] Based on the infrared spatial features and the infrared sequence features, the first classification result of the candidate location is predicted.

[0221] In one possible implementation, extracting image features from at least one of the infrared sub-images as infrared spatial features includes:

[0222] Gas scores for all the infrared sub-images are predicted respectively, wherein the gas scores are positively correlated with the number of features of leaked gas contained in the image;

[0223] The image whose gas score meets the preset screening criteria is selected from all the infrared sub-images and used as the target infrared sub-image;

[0224] The image features of the target infrared sub-image are extracted as infrared spatial features.

[0225] In one possible implementation, the system further includes a second image acquisition device, which is used to capture multiple visible light images of the target scene, wherein the time span for capturing the multiple second infrared images is the same as the time span for capturing the multiple visible light images.

[0226] The processor is further configured to extract sub-images at the candidate locations from multiple visible light images, respectively, as visible light sub-images; and to predict a second classification result of the candidate location based on all the visible light sub-images, wherein the second classification result is used to indicate that there is a moving object at the candidate location, or that there is no moving object at the candidate location;

[0227] If the second classification result indicates that there is no moving object, and the first classification result indicates that there is a gas leak, then it is determined that there is a gas leak at the candidate location.

[0228] In one possible implementation, the processor predicts a second classification result for the candidate location based on all the visible photon images, including:

[0229] Image features of the target visible photon image are extracted as visible light spatial features; and sequence features of the image sequence formed by sorting all the visible photon images in chronological order of their capture time are extracted as visible light sequence features.

[0230] Based on the visible light spatial features and the visible light sequence features, a second classification result for the candidate location is predicted;

[0231] The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

[0232] In one possible implementation, the first infrared image is obtained by the first image acquisition device capturing the target scene within a first time period preceding the time to be detected, and the second infrared image is obtained by the first image acquisition device capturing the target scene within a second time period preceding the time to be detected.

[0233] This application also provides an electronic device, as shown in FIG11, including:

[0234] Memory 1101 is used to store computer programs;

[0235] When processor 1102 executes the program stored in memory 1101, it performs the following steps:

[0236] Multiple compressed images are obtained by compressing multiple first infrared images respectively, wherein the first infrared images are uncompressed infrared images obtained by capturing the target scene;

[0237] The locations of potential gas leaks are predicted based on the multiple compressed images and used as candidate locations.

[0238] Sub-images at the candidate locations are extracted from multiple second infrared images and used as infrared sub-images, wherein the second infrared images are uncompressed infrared images obtained by capturing the target scene, and the capture time span of the multiple first infrared images is a proper subset of the capture time span of the multiple second infrared images;

[0239] A first classification result is obtained from all the infrared sub-images to predict the candidate location. The first classification result is used to indicate that there is a gas leak at the candidate location, or that there is no gas leak at the candidate location.

[0240] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0241] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0242] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described leak gas detection methods based on multi-data mixing.

[0243] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the leak gas detection methods based on multi-data mixing described above.

[0244] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0245] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0246] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system and device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0247] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for detecting a leakage gas based on multi-data mixing, characterized by, The method includes: Multiple compressed images are obtained by compressing multiple first infrared images respectively, wherein the first infrared images are uncompressed infrared images obtained by capturing the target scene; The locations of potential gas leaks are predicted based on the multiple compressed images and used as candidate locations. Sub-images at the candidate locations are extracted from multiple second infrared images and used as infrared sub-images, wherein the second infrared images are uncompressed infrared images obtained by capturing the target scene, and the capture time span of the multiple first infrared images is a proper subset of the capture time span of the multiple second infrared images; A first classification result is obtained from all the infrared sub-images to predict the candidate location. The first classification result is used to indicate that there is a gas leak at the candidate location, or that there is no gas leak at the candidate location.

2. The method of claim 1, wherein, The first classification result of the candidate location predicted based on all the infrared sub-images includes: Extract image features from at least one of the infrared sub-images as infrared spatial features; and extract sequence features from the image sequence formed by sorting all the infrared sub-images in chronological order of their capture time as infrared sequence features; Based on the infrared spatial features and the infrared sequence features, the first classification result of the candidate location is predicted.

3. The method of claim 2, wherein, Extracting image features from at least one of the infrared sub-images as infrared spatial features includes: Gas scores for all the infrared sub-images are predicted respectively, wherein the gas scores are positively correlated with the number of features of leaked gas contained in the image; The image whose gas score meets the preset screening criteria is selected from all the infrared sub-images and used as the target infrared sub-image; The image features of the target infrared sub-image are extracted as infrared spatial features.

4. The method of claim 1, wherein, The method further includes: Sub-images at the candidate locations are extracted from multiple visible light images and used as visible light sub-images. The visible light images are uncompressed visible light images obtained by capturing the target scene, and the time span of capturing the multiple second infrared images is the same as the time span of capturing the multiple visible light images. A second classification result is obtained from all the visible photon images to predict the candidate location. The second classification result is used to indicate that there is a moving object at the candidate location, or that there is no moving object at the candidate location. If the second classification result indicates that there is no moving object, and the first classification result indicates that there is a leaking gas, then it is determined that there is a leaking gas at the candidate location.

5. The method of claim 4, wherein, The second classification result for the candidate location, predicted based on all the visible photon images, includes: Image features of the target visible photon image are extracted as visible light spatial features; and sequence features of the image sequence formed by sorting all the visible photon images in chronological order of their capture time are extracted as visible light sequence features. Based on the visible light spatial features and the visible light sequence features, a second classification result for the candidate location is predicted; The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

6. The method of claim 1, wherein, The method further includes: Acquire uncompressed infrared images of the target scene captured within the first time period preceding the time to be detected, and use them as the first infrared image; An uncompressed infrared image of the target scene captured within a second time period preceding the time to be detected is obtained as a second infrared image, wherein the second time period is longer than the first time period.

7. A leak gas detection system based on multi-data mixing, characterized by, The system includes a first image acquisition device and a processor; The first image acquisition device is used to capture a target scene to obtain multiple first infrared images and multiple second infrared images, wherein the time span of capturing the multiple first infrared images is a proper subset of the time span of capturing the multiple second infrared images, and the first infrared images and the second infrared images are uncompressed infrared images; The processor is configured to predict the location of a potential leaking gas as a candidate location based on the plurality of first infrared images and the second infrared image, and to predict a first classification result of the candidate location, wherein the first classification result is used to indicate that there is a leaking gas at the candidate location, or that there is no leaking gas at the candidate location.

8. The system of claim 7, wherein, The processor predicts the location of potential gas leaks based on the plurality of first infrared images and the second infrared images, including: The plurality of first infrared images are compressed to obtain a plurality of compressed images; The location of the potential leaking gas is predicted based on the multiple compressed images; Sub-images at the candidate locations are extracted from multiple second infrared images and used as infrared sub-images; The first classification result of the candidate location is obtained based on predictions from all the infrared sub-images.

9. The system of claim 8, wherein, The first classification result of the candidate location predicted based on all the infrared sub-images includes: Extract image features from at least one of the infrared sub-images as infrared spatial features; and extract sequence features from the image sequence formed by sorting all the infrared sub-images in chronological order of their capture time as infrared sequence features; Based on the infrared spatial features and the infrared sequence features, the first classification result of the candidate location is predicted.

10. The system of claim 9, wherein, Extracting image features from at least one of the infrared sub-images as infrared spatial features includes: Gas scores for all the infrared sub-images are predicted respectively, wherein the gas scores are positively correlated with the number of features of leaked gas contained in the image; The image whose gas score meets the preset screening criteria is selected from all the infrared sub-images and used as the target infrared sub-image; The image features of the target infrared sub-image are extracted as infrared spatial features.

11. The system of claim 7, wherein, The system also includes a second image acquisition device, which is used to capture multiple visible light images of the target scene. The time span of capturing the multiple second infrared images is the same as the time span of capturing the multiple visible light images. The processor is further configured to extract sub-images at the candidate locations from multiple visible light images, respectively, as visible light sub-images; and to predict a second classification result of the candidate location based on all the visible light sub-images, wherein the second classification result is used to indicate that there is a moving object at the candidate location, or that there is no moving object at the candidate location; If the second classification result indicates that there is no moving object, and the first classification result indicates that there is a leaking gas, then it is determined that there is a leaking gas at the candidate location.

12. The system of claim 11, wherein, The processor predicts a second classification result for the candidate location based on all the visible photon images, including: Image features of the target visible photon image are extracted as visible light spatial features; and sequence features of the image sequence formed by sorting all the visible photon images in chronological order of their capture time are extracted as visible light sequence features. Based on the visible light spatial features and the visible light sequence features, a second classification result for the candidate location is predicted; The target visible photon image is the visible photon image that is far from the beginning and end of the image sequence.

13. The system of claim 7, wherein, The first infrared image is obtained by the first image acquisition device capturing the target scene within a first time period preceding the time to be detected, and the second infrared image is obtained by the first image acquisition device capturing the target scene within a second time period preceding the time to be detected.

14. An electronic device, comprising: include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.

Images