Image processing device and air conditioner

Abstract

An image processing device and an air conditioner. The image processing device comprises an acquisition unit configured to acquire a low-light image and an infrared image, and further comprises a processing unit comprising a low-light image enhancement algorithm model; the low-light image enhancement algorithm model comprises a gradient reconstruction module; the gradient reconstruction module comprises: a frequency domain processing branch configured to convert a gradient image of the low-light image from a spatial domain to a frequency domain, so as to extract a phase spectrum and an amplitude spectrum; and a spatial domain processing branch configured to fuse the gradient image of the low-light image with the infrared image in a channel dimension; and the gradient reconstruction module adds a residual of the spatial domain processing branch to the amplitude spectrum to obtain an enhanced amplitude spectrum, acquires an amplitude-enhanced gradient image of the low-light image by means of an inverse Fourier transform between the enhanced amplitude spectrum and an original phase spectrum, and generates an enhanced visible light image on the basis of a gradient feature of the amplitude-enhanced gradient image of the low-light image and an image feature of the low-light image and by means of an encoder-decoder architecture.

Description

Image processing devices and air conditioners

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese patent application No. 202411698488X, filed on November 25, 2024; and Chinese patent application No. 2024116425217, filed on November 15, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and more particularly to an image processing device and an air conditioner. Background Technology

[0004] Home appliances can utilize visible light images in various ways, primarily in areas such as surveillance and security, home control, health monitoring, entertainment and interaction, and appliance management. By acquiring visible light images and processing them, performing pattern recognition, data analysis, and control feedback, home appliances can improve their functionality and user experience. However, under low-light conditions, the quality of visible light images deteriorates, leading to reduced recognition accuracy, and impacting real-time performance and response speed.

[0005] Low-light image enhancement can employ various techniques. However, these techniques are sensitive to parameters and rely on manual adjustments, making it difficult to achieve the desired enhancement level and potentially resulting in noticeable color discrepancies. With the development of artificial intelligence, deep learning-based low-light image enhancement algorithms have emerged. However, some of these methods model the low-light domain as a normal-light domain, a process that leads to unstable enhancement results and amplifies noise. Furthermore, some methods introduce additional prior knowledge, but this prior knowledge itself is compromised, resulting in the loss of detail in the enhanced image. Summary of the Invention

[0006] Some embodiments of this application provide an image processing apparatus, including: an acquisition unit configured to acquire a low-light image and an infrared image of a target scene; and a processing unit including: a low-light image enhancement algorithm model, which includes: a gradient reconstruction module for acquiring a gradient map of the amplitude-enhanced low-light image; the low-light image enhancement algorithm model generates an enhanced visible light image based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image through an encoder-decoder architecture.

[0007] In some embodiments of this application, the gradient reconstruction module includes: a frequency domain processing branch configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; the gradient reconstruction module adds the residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains the amplitude-enhanced gradient map of the low-light image through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum.

[0008] In some embodiments of this application, the low-light image enhancement algorithm model includes: an encoder module, which includes: multiple encoder layers; each encoder layer includes: a first gradient-guided calibration unit, configured to: extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced low-light image and image features of the low-light image; generate processed image features based on the image channel importance scores and image features; generate processed gradient features based on the gradient channel importance scores and gradient features; generate main fusion features based on the processed image features and processed gradient features, and residual connections with the image features; and generate enhanced image features of the encoder layer based on the main fusion features.

[0009] In some embodiments of this application, the first gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and image features of the low-light image: performing layer normalization on the image features; performing layer normalization on the gradient features; concatenating along the channel dimension to create a combined feature representation that combines image information and gradient information; and using a linear neural network to compress the concatenated combined features at scale to extract the image feature channel importance scores of the image features and the gradient feature importance scores of the gradient features.

[0010] In some embodiments of this application, the low-light image enhancement algorithm model includes: a decoder module, which includes: multiple decoder layers, each corresponding to an encoder layer; the decoder layer is configured to: fuse the image features output by the previous decoder layer and the enhanced image features of the corresponding encoder layer to generate fused image features; the decoder layer further includes: a second gradient-guided calibration unit, which is configured to: extract image channel importance scores and gradient channel importance scores using a linear neural network based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the fused image features; generate processed fused image features based on the image channel importance scores and the fused image features; generate processed gradient features based on the gradient channel importance scores and gradient features; generate main fused features based on the processed fused image features and the processed gradient features, and the residual connection with the fused image features; generate enhanced fused image features based on the main fused features; the low-light image enhancement algorithm model generates an enhanced visible light image based on the enhanced image features generated by the decoder module.

[0011] In some embodiments of this application, the second gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and fused image features: performing layer normalization on the fused image features; performing layer normalization on the gradient features; stitching along the channel dimension to create a combined feature representation that combines image information and gradient information; and using a linear neural network to compress the stitched combined features at scale to extract the image feature channel importance scores of the image features and the gradient feature importance scores of the gradient features.

[0012] In some embodiments of this application, the low-light image enhancement algorithm model is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples correspond to the normal-light image samples, and the normal-light image samples correspond to the low-light image samples; the loss function of the gradient reconstruction module is established based on the gradient maps of the amplitude-enhanced low-light images and the gradient maps of the normal-light images generated by the gradient reconstruction module.

[0013] In some embodiments of this application, the low-light image enhancement algorithm model is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples correspond to the normal-light image samples, and the normal-light image samples correspond to the low-light image samples; the loss function of the low-light image enhancement algorithm model is established based on the enhanced visible light image and the normal-light image.

[0014] In some embodiments of this application, the loss function can be used to calculate one or more of the following: pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss.

[0015] In some embodiments of this application, the linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function.

[0016] Some embodiments of this application provide an air conditioner, including an indoor unit, comprising: an infrared imaging device capable of sampling infrared images of an air-conditioned room; a camera device capable of sampling visible light images of the air-conditioned room, the visible light images including low-light images; further comprising: a processor, the processor including: a low-light image enhancement algorithm model, which includes: a gradient reconstruction module for acquiring a gradient map of the amplitude-enhanced low-light image; the low-light image enhancement algorithm model generates an enhanced visible light image through an encoder-decoder architecture based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image; and a controller configured to perform one or more of face recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition based on the enhanced visible light image, and to control at least the operation of the indoor unit based on the recognition results.

[0017] In some embodiments of this application, the gradient reconstruction module includes: a frequency domain processing branch configured to transform the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; the gradient reconstruction module adds the residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains the amplitude-enhanced gradient map of the low-light image through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum.

[0018] Some embodiments of this application provide an image processing apparatus, including: an acquisition unit configured to acquire an image of a target scene; the image includes an infrared image and a visible light image, the infrared image and the visible light image being sampled according to a preset time sequence; and a processing unit configured to identify the emotion of a target object in the image and / or identify the emotion changes of the target object in the image within a set time period.

[0019] In some embodiments of this application, the processing unit includes: a fusion recognition model configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network, to recognize the emotion of a target object in the image and / or to recognize the emotion changes of the target object in the image within a set time period.

[0020] In some embodiments of this application, the fusion recognition model is configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network to identify the emotional changes of a target object in the image within a set time period; the processing unit further includes an emotion classification model, which is configured to identify the emotion of the target object in the image based on the image after the fusion recognition model identifies the emotional changes of the target object in the image within a set time period.

[0021] In some embodiments of this application, the fusion recognition model is configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network to identify the emotional changes of a target object in the image within a set time period; the processing unit further includes an emotion segmentation model, which is configured to generate an emotion morphology segmentation result of the target object in the image based on the image after the fusion recognition model identifies the emotional changes of the target object in the image within a set time period.

[0022] In some embodiments of this application, the fusion recognition model is configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network to identify the emotional changes of a target object in the image within a set time period; the processing unit further includes: an emotion classification model, configured to identify the emotion of the target object in the image based on the image after the fusion recognition model identifies the emotional changes of the target object in the image within a set time period; and an emotion segmentation model, configured to generate an emotion morphology segmentation result of the target object in the image based on the image after the fusion recognition model identifies the emotional changes of the target object in the image within a set time period.

[0023] In some embodiments of this application, the emotion classification model is a combination of a convolutional neural network and a Swing Transformer network; the image is sequentially passed through a convolutional neural network and a Swing Transformer network to extract encoded features, emotion classification probabilities are generated based on the encoded features, and the emotion of the target object in the image is identified based on the emotion classification probabilities.

[0024] In some embodiments of this application, the emotion segmentation model is a combination of a convolutional neural network and a Swin Transformer; the image is sequentially passed through a convolutional neural network and a Swin Transformer network to extract encoded features, and the encoded features are input into a network combination of a transposed convolutional neural network and a Swin Transformer network for decoding to generate the emotion morphology segmentation result of the target object in the image.

[0025] In some embodiments of this application, a fusion recognition model is configured to fuse infrared image feature vectors extracted by an infrared feature extraction network, visible light image feature vectors extracted by a visible light feature extraction network, and detection value feature vectors extracted by sensor detection data to recognize the emotion of a target object in an image and / or recognize the emotion changes of a target object in an image within a set time period.

[0026] In some embodiments of this application, the infrared image is added to the calculated location code and then input to the infrared feature extraction network to extract the infrared image feature vector; the visible light image is added to the calculated location code and then input to the visible light feature extraction network to extract the visible light image feature vector.

[0027] In some embodiments of this application, the fusion recognition model is configured to perform the following method to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network: concatenating or weighting the infrared image feature vectors and visible light image feature vectors to obtain fused features; applying an attention mechanism to the fused features to obtain an attention output; processing the attention output through a sigmoid function to obtain a probability vector; and recognizing the emotion of the target object in the image and / or recognizing the emotion changes of the target object in the image within a set time period based on the probability vector.

[0028] Some embodiments of this application provide an air conditioner, including: an indoor unit comprising: an infrared imaging device capable of sampling and storing infrared images of an air-conditioned room according to a preset time sequence; a camera device capable of sampling and storing visible light images of the air-conditioned room according to a preset time sequence; an indoor fan configured to supply air to the air-conditioned room; a processing unit configured to identify the emotion of a target object in the image and / or identify the emotional changes of the target object in the image within a set time period; and a control unit configured to adjust the operation of the indoor fan and / or adjust the set temperature of the air-conditioned room based on the identified emotion of the target object and / or the identified emotional changes of the target object within the set time period.

[0029] In some embodiments of this application, the processing unit includes: a fusion recognition model configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network, to recognize the emotion of a target object in the image and / or to recognize the emotion changes of the target object in the image within a set time period. Attached Figure Description

[0030] To more clearly illustrate some embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 is a schematic block diagram of the image processing apparatus provided in some embodiments of this application;

[0032] Figure 2 is a schematic block diagram of the image processing apparatus provided in some embodiments of this application;

[0033] Figure 3 is a schematic block diagram of the structure of the processing unit in an image processing apparatus provided in some embodiments of this application;

[0034] Figure 4 is a network architecture diagram of the gradient reconstruction module in an image processing apparatus provided in some embodiments of this application;

[0035] Figure 5 is a schematic block diagram of the encoder module in an image processing apparatus provided in some embodiments of this application;

[0036] Figure 6 is a schematic block diagram of the decoder module in an image processing apparatus provided in some embodiments of this application;

[0037] Figure 7 is a schematic diagram of the encoder layer in an image processing apparatus provided in some embodiments of this application;

[0038] Figure 8 is a schematic diagram of the structure of the decoder layer in an image processing apparatus provided in some embodiments of this application;

[0039] Figure 9 is a network architecture diagram of the first gradient-guided calibration unit in an image processing apparatus provided in some embodiments of this application;

[0040] Figure 10 is a network architecture diagram of the second gradient guidance calibration unit in an image processing apparatus provided in some embodiments of this application;

[0041] Figure 11 is a flowchart of the first gradient-guided calibration unit in an image processing apparatus provided in some embodiments of this application;

[0042] Figure 12 is a flowchart of the second gradient guided calibration unit in an image processing apparatus provided in some embodiments of this application;

[0043] Figure 13 is a partial training flowchart of the gradient reconstruction module in an image processing apparatus provided in some embodiments of this application;

[0044] Figure 14 is a network architecture diagram of a low-light image enhancement algorithm model in an image processing apparatus provided in some embodiments of this application;

[0045] Figure 15 is a schematic block diagram of the structure of an air conditioner provided in some embodiments of this application;

[0046] Figure 16 is a schematic block diagram of the image processing apparatus provided in some embodiments of this application;

[0047] Figure 17 is a schematic block diagram of the image processing apparatus provided in some embodiments of this application;

[0048] Figure 18 is a schematic block diagram of the image processing apparatus provided in some embodiments of this application;

[0049] Figure 19 is a schematic block diagram of the structure of the processing unit in an image processing apparatus provided in some embodiments of this application;

[0050] Figure 20 is a schematic block diagram of the structure of the processing unit in an image processing apparatus provided in some embodiments of this application;

[0051] Figure 21 is a schematic block diagram of the structure of the processing unit in an image processing apparatus provided in some embodiments of this application;

[0052] Figure 22 is a schematic block diagram of the structure of the processing unit in an image processing apparatus provided in some embodiments of this application;

[0053] Figure 23 is a flowchart of the framework of a refined emotion recognition method based on the fusion of infrared and natural images performed by an image processing apparatus provided in some embodiments of this application;

[0054] Figure 24 is a flowchart of the first stage of a refined emotion recognition method based on the fusion of infrared and natural images performed by an image processing apparatus provided in some embodiments of this application.

[0055] Figure 25 is a flowchart of the framework of a refined emotion recognition method based on the fusion of infrared and natural images performed by an image processing apparatus provided in some embodiments of this application.

[0056] Figure 26 is a network architecture diagram of the input layer of the infrared feature extraction network in an image processing apparatus provided in some embodiments of this application;

[0057] Figure 27 is a network architecture diagram of the intermediate layer of the infrared feature extraction network in an image processing apparatus provided by some embodiments of the invention;

[0058] Figure 28 is a network architecture diagram of the output layer of the infrared feature extraction network in an image processing apparatus provided by some embodiments of the invention;

[0059] Figure 29 is an architecture diagram of an improved Xception network that can be used in the fusion recognition module of an image processing apparatus provided in some embodiments of the invention;

[0060] Figure 30 is a flowchart of the fusion recognition module in the image processing apparatus provided in some embodiments of the invention.

[0061] In the diagram: 1. Image processing device; 10. Acquisition unit; 20. Processing unit; 201. Central processing unit; 202. Read-only memory; 203. Random access memory; 204. Input / output interface; 205. Bus; 211. Fusion recognition model; 212. Emotion classification model; 213. Emotion segmentation model; 214. Infrared feature extraction network; 215. Visible light feature extraction network; 30. Low-light image enhancement algorithm model; 40. Storage unit; 301. Gradient reconstruction module; 302. Encoder module; 303. Decoder module; 401. Frequency domain processing branch; 402. Spatial domain processing branch; 501. Encoder layer; 502. Decoder layer; 601. First gradient guidance calibration unit; 602. Second gradient guidance calibration unit; 100. Air conditioner; 71. Processor; 72. Controller. Detailed Implementation

[0062] The technical solutions of some embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on some embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0063] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0064] The terms "first" and "second" are used descriptively only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0065] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0066] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include contact between the first and second features, or contact between the first and second features through other features between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0067] The following disclosure provides many different embodiments or examples for implementing different structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. In addition, various specific examples of processes and materials are provided in this application, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0068] Figure 1 illustrates the functional configuration of the image processing apparatus 1 disclosed in this application.

[0069] In some embodiments of this application, the image processing apparatus 1 includes an acquisition unit 10 and a processing unit 20. In some embodiments of this application, the acquisition unit 10 may be integrated into the image processing apparatus 1. In some embodiments of this application, the acquisition unit 10 may also be connected to the image processing apparatus 1 via wired or wireless communication. In some embodiments of this application, the acquisition unit 10 is configured to acquire an image of a target scene. The image includes infrared images and visible light images, with the visible light image including a low-light image. In some embodiments of this application, the infrared image and the visible light image are acquired by an infrared imaging device and a camera device, respectively.

[0070] Infrared imaging devices utilize infrared radiation for imaging. All objects emit infrared radiation of different wavelengths depending on their temperature; the higher the temperature, the greater the radiation intensity and the shorter the wavelength. Infrared imaging devices detect infrared wavelength signals and convert them into different electrical signals to capture thermal energy, further converting it into images (or videos) for observation and analysis. The images output by infrared imaging devices are typically called infrared images or thermal images, where different colors represent different temperatures. For example, red or white can represent heat sources, while blue represents cooler areas. Camera devices, on the other hand, capture visible light wavelengths to create images, converting light signals into image signals to record the corresponding scene. Camera devices are imaging devices based on related technologies. In some embodiments of this application, the infrared imaging device and the camera device have a viewing angle covering the same target scene. The viewing angles of the infrared imaging device and the camera device can be fixed or adjustable.

[0071] As shown in Figure 2, the processing unit 20 includes a central processing unit 201, a read-only memory 202 (ROM), and a random access memory 203 (RAM). The central processing unit 201 executes various processing tasks based on programs stored in the read-only memory 202 or programs loaded from storage units into the random access memory 203. Furthermore, the random access memory 203 appropriately stores data required by the central processing unit 201 to execute these tasks. The central processing unit 201, the read-only memory 202, and the random access memory 203 are interconnected via a bus 205. An input / output interface 204 is also connected to the bus 205.

[0072] As shown in Figure 3, the processing unit 20 is equipped with a low-light image enhancement algorithm model 30. This model uses image processing technology to improve the quality of images captured under low-light conditions. By extracting and enhancing the gradient features of the low-light image and combining this with information from the infrared image, it generates a higher-quality visible light image. The image processing device 1 is particularly suitable for use with home appliances to improve their performance under low-light conditions, such as baby monitors, scene lighting, and refrigerators.

[0073] In some embodiments of this application, as shown in Figures 3 and 4, the low-light image enhancement algorithm model 30 includes a gradient reconstruction module 301. The gradient reconstruction module 301 is a part of the low-light image enhancement algorithm model 30, extracting and enhancing the gradient features of the low-light image through frequency and spatial domain processing methods. In some embodiments of this application, the gradient reconstruction module 301 includes a frequency domain processing branch 401 and a spatial domain processing branch 402. In some embodiments of this application, the frequency domain processing branch 401 is configured to process the gradient map I of the low-light image... gradient_lowThe spatial domain is transformed to the frequency domain (via Fourier transform, FFT) to extract the phase and amplitude spectra of the low-light image. The phase spectrum contains structural information of the low-light image, while the amplitude spectrum contains intensity information. In some embodiments of this application, the spatial domain processing branch 402 is configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension, utilizing thermal radiation information in the infrared image to supplement and enhance the details of the low-light image.

[0074] In some embodiments of this application, the gradient reconstruction module 301 adds the residual of the spatial domain processing branch 402 to the amplitude spectrum output by the frequency domain processing branch 401 to obtain an enhanced amplitude spectrum; through the inverse Fourier transform (IFFT) between the enhanced amplitude spectrum and the original phase spectrum, the gradient map I of the amplitude-enhanced low-light image is obtained. gradient_enhance In some embodiments of this application, the low-light image enhancement algorithm model 30 generates an enhanced visible light image based on the gradient features of the gradient map of the low-light image with amplitude enhancement, and the image features of the low-light image, through an encoder-decoder architecture. The encoder extracts the features of the image, and the decoder reconstructs the enhanced image from the features.

[0075] In some embodiments of this application, during the gradient feature enhancement process of an image, data from more modalities (e.g., radar images, depth images) can be introduced. Through a weighting strategy using deep learning or a fusion network, image information from different modalities can be effectively fused, thereby improving the image enhancement effect under low-light conditions. An adaptive modal fusion strategy based on image quality assessment is developed. Under different environmental conditions (such as different light intensities or environmental complexity), this strategy dynamically adjusts the weights of each modal image to improve the quality of the enhanced image.

[0076] The low-light image enhancement algorithm model 30 provided in some embodiments of this application introduces gradient prior knowledge containing rich detail texture information and performs modal fusion with an infrared image that also has detailed features to reconstruct gradient feature information that is closer to that of a normal-light image. This enables the acquisition of more complete image detail structure features and ensures that no detail loss occurs during the entire low-light image enhancement process.

[0077] The encoder-decoder architecture can adopt the U-Net architecture or other related technical architecture designs. The specific design of the encoder-decoder will be described in detail below. As shown in Figures 3 and 5, in some embodiments of this application, the encoder-decoder architecture includes an encoder module 302, which includes multiple encoder layers 501. As shown in Figures 3 and 6, in some embodiments of this application, the encoder-decoder architecture includes a decoder module 303, which includes multiple decoder layers 502.

[0078] In some embodiments of this application, multiple encoder layers 501 and multiple decoder layers 502 are symmetrically designed. The multiple encoder layers 501 progressively downsample the image to extract features, while the multiple decoder layers 502 progressively upsample the image to reconstruct it. The multiple decoder layers 502 employ skip connections, which pass the features output by the encoder layers 501 to the corresponding decoder layers 502, helping the decoder layers 502 recover detailed information. In some embodiments of this application, the encoder layer 501 includes a first gradient-guided calibration unit 601.

[0079] In some embodiments, a self-supervised loss function, such as local consistency loss, structural loss, or texture loss, is added to the encoder-decoder architecture. This self-supervised learning strategy enables the model to learn enhancement rules for low-light images even without a large amount of labeled data. Introducing a discriminator from a Generative Adversarial Network (GAN) further improves the visual consistency between the generated enhanced image and the original real image, thereby enhancing the enhancement results.

[0080] As shown in Figures 7 and 9, the first gradient-guided calibration unit 601 is configured as follows:

[0081] Based on the gradient features of the gradient map of the amplitude-enhanced low-light image and the image features of the low-light image, a linear neural network is used to extract the image channel importance score and the gradient channel importance score.

[0082] Image features are generated based on image channel importance scores and image features;

[0083] Gradient features are generated based on gradient channel importance scores and gradient features after processing;

[0084] The main fusion features are generated based on the processed image features and the processed gradient features, as well as the residual connections with the image features.

[0085] Enhanced image features of encoder layer 501 are generated based on the main fusion features.

[0086] The first gradient-guided calibration unit 601 in encoder module 302 uses gradient feature information to guide the image feature enhancement process; throughout the processing of encoder layer 501, gradient feature information is integrated into the image feature enhancement steps to ensure that the enhanced image features of the final encoder layer 501 can better reflect the details and structural information of the image.

[0087] In some embodiments of this application, the first gradient-guided calibration unit 601 is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and image features of the low-light image:

[0088] Layer normalization is performed on image features;

[0089] Layer normalization is applied to the gradient features;

[0090] The data is stitched along the channel dimension to create a combined feature representation that combines image information and gradient information;

[0091] A linear neural network is used to compress the stitched combined features at scale to extract the image feature channel importance scores of image features and the gradient feature importance scores of gradient features.

[0092] By performing layer normalization on image features and gradient features separately and concatenating them along the channel dimension, a combined feature representation combining image and gradient information is created. Then, a linear neural network is used to compress the concatenated combined features at scale, extracting the channel importance scores of image features and gradient features. This effectively fuses image and gradient information, generating more representative channel importance scores based on stable and accurate features, resulting in enhanced image features of encoder layer 501 with higher quality.

[0093] In some embodiments, after concatenating image and gradient features, multiple convolutional layers or deep learning layers can be introduced to fuse the multi-level abstract information of image and gradient features layer by layer. This multi-level feature fusion method helps capture detailed information from local to global perspectives, while enhancing the correlation between high-level semantic information and low-level details. Based on the multi-dimensional concatenation of image and gradient features, a multi-channel fusion module can be set up. This module not only performs concatenation in the channel dimension but also cross-fusion in the spatial dimension, so that gradient information not only guides the extraction of channel importance but also affects the feature enhancement of each pixel.

[0094] As shown in Figures 8 and 10, the decoder layer 502, based on skip connections, fuses the image features output by the previous decoder layer with the enhanced image features of the corresponding encoder layer 501 to generate fused image features; the decoder layer 502 also includes a second gradient-guided calibration unit 602. The architecture of the second gradient-guided calibration unit 602 is similar to that of the first gradient-guided calibration unit 601.

[0095] The second gradient-guided calibration unit 602 is configured as follows:

[0096] Based on gradient features of gradient maps of amplitude-enhanced low-light images and fused image features, a linear neural network is used to extract image channel importance scores and gradient channel importance scores.

[0097] The fused image features are generated based on the importance scores of image channels and the fused image features.

[0098] Gradient features are generated based on gradient channel importance scores and gradient features after processing;

[0099] The main fusion features are generated based on the processed fused image features and the processed gradient features, as well as the residual connections with the fused image features.

[0100] Generate enhanced fused image features based on key fusion features;

[0101] The low-light image enhancement algorithm model 30 generates an enhanced visible light image based on the enhanced image features generated by the decoder module 303.

[0102] By introducing a first gradient-guided calibration unit 601 and a second gradient-guided calibration unit 602 in the encoder layer 501 and decoder layer 502 respectively, the image feature enhancement process is guided by gradient feature information, ensuring the effective extraction and preservation of image details and structural information. The second gradient-guided calibration unit 602 in decoder layer 502 further fuses image and gradient information based on skip connections, generating higher-quality enhanced image features. This dual-guided mechanism not only improves the stability and accuracy of feature extraction but also ensures that the final enhanced visible light image has higher clarity and detail richness under low-light conditions.

[0103] In some embodiments, the second gradient-guided calibration unit 602 is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced, low-light image and fused image features: performing layer normalization on the fused image features; performing layer normalization on the gradient features; stitching along the channel dimension to create a combined feature representation that combines image information and gradient information; and using a linear neural network to compress the stitched combined features at scale to extract the image feature channel importance scores of the image features and the gradient feature importance scores of the gradient features.

[0104] The second gradient-guided calibration unit 602 in the decoder layer 502 extracts the channel importance score of the fused image features and gradient features through layer normalization, feature splicing and linear neural network compression, thereby generating more representative enhanced image features and ensuring that the final enhanced image features can better reflect the details and structural information of the image.

[0105] In some embodiments of this application, an image quality assessment module is provided to evaluate the quality of the enhanced image in real time based on a deep learning model, scoring indicators such as brightness, contrast, and sharpness. This provides real-time feedback to guide the second gradient-guided calibration unit 602 in adjusting parameters during the enhancement process. Based on the quality assessment results, the weighting coefficients of the image and gradient features, or the feature extraction method, are dynamically adjusted. For example, under low-light conditions, image details may be insignificant, so the weight of gradient features can be increased; while under strong light conditions, the weight of gradient features can be appropriately reduced to improve the sharpness of the enhanced image. A temporal convolutional neural network (TCN) or a long short-term memory network (LSTM) is introduced into the second gradient-guided calibration unit 602 to adjust the image enhancement strategy of the current frame based on the gradient features and image features of previous and subsequent frames. This reduces inconsistencies in dynamic scenes and enhances the consistency of image details at different time points.

[0106] In some embodiments of this application, the low-light image enhancement algorithm model 30 is trained based on a dataset, which includes low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples correspond to the normal-light image samples, and vice versa. In some embodiments of this application, the loss function of the gradient reconstruction module 301 is established based on the gradient maps of the amplitude-enhanced low-light images and normal-light images generated by the gradient reconstruction module 301. In some embodiments of this application, the loss function of the low-light image enhancement algorithm model 30 is established based on the enhanced visible light images and normal-light images.

[0107] By training the model using a dataset containing low-light image samples, normal-light image samples, and infrared image samples, the model is ensured to learn effective image enhancement features under various lighting conditions. In some embodiments of this application, by establishing a loss function based on enhanced visible light images and normal-light images, the model can directly optimize the difference between the enhanced image and the target image, thereby improving the quality and accuracy of the enhanced image. This method not only utilizes multimodal data (such as infrared images) to supplement the information in low-light images, but also ensures that the model generates high-quality enhanced images under low-light conditions through effective loss function design.

[0108] In some embodiments of this application, the loss function can be used to calculate one or more of pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss. By comprehensively considering multiple loss functions, including pixel consistency loss, perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss, the quality of the enhanced image can be comprehensively evaluated and optimized. This combination of multiple loss functions not only ensures the accuracy of the enhanced image at the pixel level but also considers the image's perceptual quality, frequency domain characteristics, edge details, and gradient information, thereby generating more realistic and detailed enhanced images. Through this comprehensive optimization strategy, the model can generate high-quality enhanced visible light images under low-light conditions, significantly improving image clarity and visual effects.

[0109] In some embodiments of this application, the linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function. By using a linear neural network composed of global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function, feature information can be effectively extracted and compressed. Global average pooling compresses spatial dimension information into channel dimension information. The first linear layer and the ReLU activation function introduce non-linearity, increasing the complexity of feature representation. The second linear layer and the Sigmoid activation function compress the features to between 0 and 1, generating channel importance scores. This design is not only concise and efficient but also accurately evaluates and adjusts the importance of image and gradient features, thereby generating more representative enhanced image features and improving the image enhancement effect. In some embodiments, ReLU (Rectified Linear Unit) is an activation function applied in neural networks, especially in deep learning. The mathematical expression for ReLU is: f(x) = max(0,x), meaning that if the input value x is greater than 0, the output is equal to x; if x is less than or equal to 0, the output is 0. The Sigmoid activation function is a non-linear activation function that can be used in the output layer of binary classification problems, as well as in the hidden layers of neural networks. Its mathematical expression is: The output range of the Sigmoid activation function is (0,1), meaning that regardless of whether the input value x is positive or negative, the output will always be between 0 and 1.

[0110] In some embodiments of this application, the image processing apparatus 1 executes a low-light image enhancement algorithm guided by infrared information, the network architecture of which is shown in Figure 14. In some embodiments of this application, the low-light image enhancement algorithm guided by infrared information is divided into multiple stages. These stages are described below.

[0111] Phase 1: Generating the dataset.

[0112] In some embodiments of this application, image data samples in the dataset are acquired using visible light imaging devices and infrared imaging devices. In some embodiments of this application, the imaging resolution of the visible light imaging device is 1024×768 pixels. In some embodiments of this application, the imaging resolution of the infrared imaging device is 640×512 pixels. In some embodiments of this application, the dataset includes visible light images and infrared images. In some embodiments of this application, the visible light images further include low-light image samples and normal-light image samples; the low-light image samples and normal-light image samples correspond to each other, with each low-light image sample corresponding to one normal-light image sample. The low-light image samples and normal-light image samples have the same content but different brightness. Low-light image samples refer to images taken in low-light environments, while normal-light image samples refer to images taken in well-lit environments. In some embodiments of this application, corresponding low-light image samples and normal-light image samples form data pairs. In some embodiments of this application, low-light image samples and normal-light image samples can be distinguished by brightness values. For example, when the average brightness value is lower than a preset brightness threshold, it is considered a low-light image sample; while when the average brightness value is within a certain preset range, it is considered a normal-light image sample. In some embodiments of this application, infrared images include infrared image samples. In some embodiments of this application, infrared image samples are acquired simultaneously when acquiring low-light image samples; low-light image samples and infrared image samples correspond to each other, with each low-light image sample corresponding to one infrared image sample. In some embodiments of this application, corresponding low-light image samples and infrared image samples form data pairs. In some embodiments of this application, the dataset is processed through manual data filtering to remove data pairs with large deviations.

[0113] In some embodiments of this application, the infrared image samples are expanded to 1024×768 pixels and aligned with the visible light image samples using a bilinear interpolation method. Bilinear interpolation is used to generate new pixel values ​​when the image is scaled or deformed. The target pixel value is calculated by considering the values ​​of the four nearest surrounding pixels, thus achieving a smooth image transformation. In some embodiments of this application, the imaging resolution of the infrared imaging device is 640×512 pixels, different from the imaging resolution of the visible light imaging device. To align the infrared image samples with the visible light image samples, the infrared image samples are adjusted to the same resolution as the visible light image samples. The bilinear interpolation method is an algorithm in related technologies and will not be elaborated upon here. Multiple sets of filtered data pairs are used as the final dataset, which is then divided into a training set and a test set. The training set is used for training and optimizing the algorithm, while the test set is used for performance evaluation. The ratio of the training set to the test set is determined according to actual needs, ensuring data independence and representativeness between them, thereby more accurately reflecting the actual effect of the algorithm.

[0114] Phase 2: Constructing a low-light image enhancement algorithm model 30.

[0115] In some embodiments of this application, the low-light image enhancement algorithm model 30 is a low-light image enhancement algorithm model based on infrared image integration. In some embodiments of this application, the low-light image enhancement algorithm model 30 includes: a gradient reconstruction module 301, an encoder module 302, and a decoder module 303. In some embodiments of this application, the encoder module 302 includes a first gradient-guided calibration unit 601. In some embodiments of this application, the decoder module 303 includes a second gradient-guided calibration unit 602.

[0116] First, a gradient map of the low-light image is generated. A gradient map of a low-light image is an image generated by calculating the gradients of pixel intensity changes in a low-light image. The gradient map of a low-light image highlights edge and detail information; specifically, the horizontal and vertical gradients of the low-light image can be calculated using convolution operations with the Sobel operator, and further, the gradient magnitude and direction are calculated using the square root of the sum of the squares of the horizontal and vertical gradients and the arctangent.

[0117] The gradient map of a low-light image is contaminated with numerous noise points, which can damage the context and detail information within the image. Due to inaccurate gradient priors, residual noise or artifacts may exist in the enhanced low-light image. In some embodiments of this application, a hybrid interactive method combining spatial and frequency domains is proposed to compensate for the damaged gradient map of a low-light image. The spatial domain of a low-light image refers to how it is represented in its original pixel coordinate system; while in the frequency domain, the low-light image is represented as a superposition of different frequency components, which reflect different structures and features within the image. On one hand, the spatial domain of a low-light image contains structural integrity and rich semantic information, forming the basis for visual understanding. On the other hand, the phase spectrum in the frequency domain of a low-light image primarily captures its structural essence, while the amplitude spectrum encodes semantic nuances including brightness levels and noise components. Furthermore, infrared low-light images are inherently insensitive to light and can clearly reflect the edge contours of objects, including a wealth of structural information. This structural information can assist in the reconstruction of the low-light image gradient.

[0118] In some embodiments of this application, a multi-stage image enhancement process is set up, with each stage progressively improving the brightness, contrast, and sharpness of the image to avoid artifacts or distortion caused by over-enhancement. At each stage, fine-grained adjustments are made to continuously improve the details of the low-light image, generating a higher-quality enhanced image. A denoising network is introduced to remove noise from the gradient map and the enhanced image. Combining structural information from the infrared image effectively distinguishes noise from meaningful details, reducing noise interference during low-light image enhancement. The low-light image is divided into multiple layers (such as texture, edge, and brightness layers), and each layer is enhanced independently. In this way, the brightness, contrast, and details of different elements in the image can be better adjusted, resulting in a more realistic and clearer final image. Based on deep learning image analysis, a content-aware enhancement strategy is set up, enabling regions with specific content in the image to be automatically identified and enhanced accordingly. For example, object edges and facial features receive more detail enhancement, while background areas maintain a lower enhancement level to avoid over-processing.

[0119] In some embodiments of this application, the gradient reconstruction module 301 is used to adjust the amplitude spectrum using global feature information while maintaining the integrity of the image structure. This method effectively denoises and enhances the semantic importance of the gradient image, thus providing more robust information for the low-light image enhancement process. Figure 4 shows the network architecture of the gradient reconstruction module 301. In some embodiments of this application, the gradient reconstruction module 301 includes a frequency domain processing branch 401 and a spatial domain processing branch 402. The frequency domain processing branch 401 transforms the gradient map of the low-light image from the spatial domain to the frequency domain, extracting the phase spectrum and amplitude spectrum of the low-light image. The spatial domain processing branch 402 fuses the gradient map of the low-light image with the infrared image in the channel dimension to extract rich texture and structural information. The gradient reconstruction module 301 adds the residual from the spatial domain processing branch 402 to the amplitude spectrum to obtain an enhanced amplitude spectrum. Through the inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum, the gradient map of the low-light image with enhanced amplitude is obtained.

[0120] In some embodiments of this application, the spatial domain processing branch 402 includes a 1×1 convolutional layer with a Leaky ReLU activation function to increase the dimension of the gradient map of the low-light image; after the 1×1 convolutional layer with a Leaky ReLU activation function, grouped convolution is used: a 5×5 convolutional layer with four groups and a Leaky ReLU activation function is set. In this case, the input channels are divided into four groups, and each group performs a 5×5 convolution operation independently to extract spatial information and maintain a certain expressive power while reducing the number of parameters and computational complexity. Finally, the dimension is reduced by a 1×1 convolutional layer, and the residual of the amplitude spectrum is obtained by a sigmoid activation function.

[0121] The above process can be represented by the following formula: A,P = FFT(I gradient_low ), A res =Function(Concat(I gradient_low ,I infrared )), I gradient_enhance =IFFT(A+A) res ,P),

[0122] in:

[0123] A and P are the amplitude and phase spectra of the gradient map of the low-light image, respectively, and FFT(*) represents the Fourier transform; I gradient_low Represents the gradient map of a low-light image; A,P = FFT(I gradient_low ) indicates that a Fourier transform is performed on the gradient map of the low-light image to obtain the frequency domain representation of the low-light image. The frequency domain representation of the low-light image includes the amplitude spectrum and the phase spectrum.

[0124] Iinfrared Represents an infrared image, Concat(I gradient_low ,I infrared ) represents the gradient map I of a low-light image. gradient_low and I infrared Perform concatenation along the channel dimension; Function(Concat(I gradient_low ,I infrared This indicates that a series of convolution operations are performed on the stitched image, including dimensionality enhancement, spatial information extraction, and dimensionality reduction, ultimately yielding the residual A of the amplitude spectrum. res A res The residual between the amplitude of the gradient map of a low-light image and the ideal amplitude reflects the insufficiency or deviation of the gradient map of the low-light image in the amplitude spectrum.

[0125] IFFT stands for Inverse Fourier Transform, I gradient_enhance Gradient plot representing an enhanced low-light image.

[0126] The gradient reconstruction module 301 effectively utilizes a limited number of convolutional layers to fully leverage the inherent semantic and structural information in the original image, minimizing information loss during denoising and enhancement processes and ensuring high-quality results.

[0127] Encoder module 302 is used to progressively compress gradient features and extract high-level abstract features. In some embodiments of this application, encoder module 302 includes three encoder layers 501. Each encoder layer processes features at a different scale. Extracting features at different scales allows encoder module 302 to capture different levels of contextual information in the image. Shallower encoder layers focus more on local details, while deeper encoder layers can capture broader contextual information; for example, the first layer extracts low-level features, such as edges; the second layer extracts mid-level features, such as shapes; and the third layer extracts high-level features, such as contextual information. Therefore, the scale of the feature map changes with each encoder layer 501: the width is halved, the height is halved, and the number of channels is doubled. Each encoder layer 501 includes a first gradient guidance calibration unit 601. The image features X obtained after a 3×3 convolution operation on a low-light image are... in The gradient map of the amplitude-enhanced, low-light image is fed into encoder layer 501 as one input; the gradient feature G is obtained after a 3×3 convolution operation. in It is fed into encoder layer 501 as another input. In the first gradient-guided calibration unit 601, several steps as shown in FIG11 are performed.

[0128] Step S101: Process image features X in Layer normalization is performed, and the gradient feature G is also processed.in Perform layer normalization.

[0129] Step S102: Concatenate along the channel dimension to create a combined feature representation that combines image information and gradient information.

[0130] Step S103: Use a linear neural network to compress the stitched combined features at different scales to extract image features X. in Image feature channel importance scores and gradient features G in Gradient feature channel importance score.

[0131] The forward propagation process of a linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a sigmoid activation function. Global average pooling performs a spatial averaging on each channel, converting the concatenated combined features into a scalar. The first linear layer maps the feature vector after global average pooling to a new feature space, resulting in a new vector. The ReLU activation function performs element-wise operations on the output of the first linear layer, learning complex feature representations and adding non-linearity. The second linear layer and the sigmoid function perform element-wise operations on the output of the second linear layer, compressing the channel importance scores to the range [0,1], where the score of each channel represents its importance. Based on the output of the sigmoid function, image features X are further extracted. in Image feature channel importance scores and gradient features G in The gradient feature channel importance scores (e.g., through segmentation operations, which will be explained below) effectively identify channels that contain information that is crucial for image enhancement.

[0132] In some embodiments of this application, the loss function and optimization algorithm used in the backpropagation process of the linear neural network can be algorithms from related technologies, such as using the cross-entropy loss function as the loss function and stochastic gradient descent, Adam, etc., as optimization algorithms. The true labels of the samples used to train the linear neural network can be generated using prediction algorithm models from related technologies, such as SHAP, LIME, or other related prediction models, or they can be generated based on pre-configured rules. In some embodiments, Adam (Adaptive Moment Estimation) is an optimization algorithm, especially in deep learning, which combines the advantages of gradient descent with the mechanism of adaptive learning rate. SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations) are two model interpretation methods designed to help understand the prediction results of complex black-box models (such as ensemble models like deep learning and random forests).

[0133] The above-mentioned steps of the first gradient-guided calibration unit 601 can be expressed by the following formula: U = Concat(LayerNorm(X) in G in )) σ u =Linear(ReLU(Linear(GAP(U))))

[0134] Where LayerNorm represents layer normalization, Concat represents concatenating the normalized features along the channel dimension, U represents the combined features after concatenation, GAP represents global average pooling, Linear represents a linear layer, ReLU is the ReLU activation function, and σ u This is the output of the second linear layer.

[0135] Step S104: Compare the importance scores of image feature channels with image feature X. in Element-wise multiplication is performed to obtain the processed image features; the gradient feature channel importance scores are then compared with the gradient feature G. in The processed gradient features are obtained by performing element-wise multiplication.

[0136] Step S105: Add the processed image features and the processed gradient features element-wise. That is, fuse the processed image features and the processed gradient features by adding them element-wise, combining the enhanced image features with the fine spatial details captured by the gradient features to obtain a more comprehensive representation.

[0137] Step S106: Add the processed image features and the processed gradient features element by element to the image feature X. in Residual connections are established, and their outputs are primarily fused features S. For example, residual connections can be established through element-wise addition. This residual connection preserves the original image features, allowing the linear neural network to learn residual enhancements rather than the full transformation. The primary fused feature S combines image and gradient information and emphasizes channels crucial for effective enhancement.

[0138] The aforementioned steps of the first gradient-guided calibration unit 601 can be expressed by the following formula: σ1,σ2=Split(Sigmoid(σ u S=(1+σ1)×X in +σ2×G in

[0139] In the above formula, σ u For the output of the second linear layer, Sigmoid(σ) u The first step () applies the Sigmoid activation function to the output of the second linear layer, mapping the input to the (0,1) interval. The second step () is a segmentation operation (also called a partitioning operation), which divides the input into two parts, σ1 and σ2, based on the output of the Sigmoid function. σ1 represents the importance score of the image feature channels, and σ2 represents the importance score of the gradient feature channels. The importance score of each channel reflects its contribution. The segmentation operation can allocate the calculated "channel importance scores" according to their original sources, namely image features and gradient features, thus distinguishing their contributions.

[0140] Further, the importance score σ1 of the image feature channel is compared with the image feature X. in The processed image features are obtained by performing element-wise multiplication, i.e., σ1×X. in The gradient feature channel importance score σ² is compared with the gradient feature G. in Element-wise multiplication yields the processed gradient feature, i.e., σ²×G. in The processed image features and the processed gradient features are added element-wise, resulting in σ1×X. in +σ2×G in The result of element-wise summing the processed image features and the processed gradient features is compared with the image feature X. in Establishing a residual connection yields σ1×X in +σ2×G in +X in Thus, we obtain S = (1 + σ1) × X in +σ2×G in

[0141] Among them, S is the main fusion feature.

[0142] Step S107: Further perform a three-layer convolution operation on the main fusion features.

[0143] The main fusion feature S is first normalized, then passed through a 3×3 convolution, a 1×1 convolution, a Leaky ReLU activation function, and another 1×1 convolution. It is then identity-connected to the main fusion feature S to obtain the output X of the first gradient-guided calibration unit 601. out To enrich the texture information in the image, the output X of the first gradient-guided calibration unit 601 out This corresponds to the enhanced image features of encoder layer 501. In some embodiments, Leaky ReLU (Leaky Rectified Linear Unit) is a variant of the ReLU activation function, used to address some shortcomings of the ReLU activation function, especially the "dead neuron" problem. In the ReLU activation function, the output is always 0 when the input is less than zero, which may cause some neurons in the neural network to lose the opportunity to update during training, resulting in the model failing to learn effectively. The Leaky ReLU activation function avoids this problem by allowing a small slope for negative inputs.

[0144] Corresponding to encoder module 302, the decoder progressively expands the low-dimensional feature representation into a high-dimensional image. Decoder module 303 includes three decoder layers 502. Each decoder layer 502 corresponds to one encoder layer 501. In each decoder layer 502, the enhanced image feature maps in the corresponding encoder layer 501 are directly connected via skip connections. Each decoder layer 502 includes a second gradient-guided calibration unit 602. The network architecture of the second gradient-guided calibration unit 602 is basically the same as that of the first gradient-guided calibration unit 601.

[0145] In some embodiments of this application, multiple steps as shown in FIG12 are performed in each decoder layer 502. For the first decoder layer 502, the enhanced image features X corresponding to the encoder layer 501 are processed. out and gradient features G in As the input to the second gradient-guided calibration unit 602, the operation of the second gradient-guided calibration unit 602 for the first decoder layer 502 is similar to that of the first gradient-guided calibration unit 601, and will not be repeated here.

[0146] For other decoder layers 502, the previous layer is executed first, and the image features output by the decoder layer and the enhanced image features X of the corresponding encoder layer 501 are processed. outThe fusion is performed on the fused features, and the output is used as one input to the second gradient-guided calibration unit 602 in the decoder layer 502. This is denoted as the image feature of the second gradient-guided calibration unit 602 in the encoder layer 501, i.e., the fused image feature X′. in Gradient feature G in It serves as another input to the second gradient-guided calibration unit 602 in the decoder layer 502.

[0147] In the second gradient-guided calibration unit 602 of the decoder layer 502, the following steps are performed:

[0148] Step S201: Analyze the fused image features X′ in Layer normalization is performed, and the gradient feature G is also processed. in Perform layer normalization.

[0149] Step S202: Concatenate along the channel dimension to create a combined feature representation that combines image information and gradient information;

[0150] Step S203: Use a linear neural network to compress the stitched combined features at scale to extract image features X′. in Image feature channel importance scores and gradient features G in Gradient feature channel importance score.

[0151] The forward propagation process of a linear neural network includes global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a sigmoid activation function. Global average pooling performs a spatial averaging on each channel, converting the concatenated combined features into a scalar. The first linear layer maps the feature vector after global average pooling to a new feature space, resulting in a new vector. The ReLU activation function performs element-wise operations on the output of the first linear layer, learning complex feature representations and adding non-linearity. The second linear layer and the sigmoid function perform element-wise operations on the output of the second linear layer, compressing the channel importance scores to the range [0,1], where the score of each channel represents its importance. Based on the output of the sigmoid function, image features X are further extracted. in Image feature channel importance scores and gradient features G in The gradient feature channel importance scores (e.g., through segmentation operations, which will be explained below) effectively identify channels that contain information that is crucial for image enhancement.

[0152] In some embodiments of this application, the loss function and optimization algorithm used in the backpropagation process of the linear neural network can be algorithms from related technologies, such as using the cross-entropy loss function as the loss function and stochastic gradient descent, Adam, etc. as optimization algorithms. The true labels of the samples used to train the linear neural network can be generated using prediction algorithm models from related technologies, such as SHAP, LIME, or other known prediction models, or can be generated based on pre-configured rules.

[0153] The above-mentioned steps of the second gradient guided calibration unit 602 can be expressed by the following formula:

[0154] U′=Concat(LayerNorm(X′ in G in ))

[0155] σ′ u =Linear(ReLU(Linear(GAP(U′))))

[0156] Where LayerNorm represents layer normalization, Concat represents concatenating the normalized features along the channel dimension, U′ represents the combined features after concatenation; GAP represents global average pooling; Linear represents a linear layer; ReLU is the ReLU activation function; σ′ u This is the output of the second linear layer.

[0157] Step S204: Compare the importance scores of image feature channels with image feature X′ in Element-wise multiplication is performed to obtain the processed image features; the gradient feature channel importance scores are then compared with the gradient feature G. in The processed gradient features are obtained by performing element-wise multiplication.

[0158] Step S205: Add the processed image features and the processed gradient features element-wise. That is, fuse the processed image features and the processed gradient features by adding them element-wise, combining the enhanced image features with the fine spatial details captured by the gradient features to obtain a more comprehensive representation.

[0159] Step S206: Add the processed image features and the processed gradient features element by element to the image feature X′. in Residual connections are established, and their outputs primarily fuse the feature S′. For example, residual connections can be established through element-wise addition. This residual connection preserves the original image features, allowing the linear neural network to learn residual enhancements rather than the full transformation. The primary fusion feature S combines image and gradient information and emphasizes channels crucial for effective enhancement.

[0160] The above-mentioned steps of the second gradient guided calibration unit 602 can be expressed by the following formula: σ′1,σ′2=Split(Sigmoid(σ′ u S′=(1+σ′1)×X′ in +σ′2×G in

[0161] In the above formula, σ′ u The output of the second linear layer, Sigmoid(σ′) u The first step () applies the Sigmoid activation function to the output of the second linear layer, mapping the input to the (0,1) interval. The second step () is a segmentation operation, which divides the input into two parts, σ′1 and σ′2, based on the output of the Sigmoid function. σ′1 represents the importance score of the image feature channels, and σ′2 represents the importance score of the gradient feature channels. For example, the segmentation operation can make the sum of σ′1 and σ′2 equal to 1. It can also be attention-based segmentation, statistical feature-based segmentation, learning-based segmentation, or even equal division.

[0162] Further, the image feature channel importance score σ′1 is compared with the image feature X′. in The processed image features are obtained by performing element-wise multiplication, i.e., σ′1×X′. in The gradient feature channel importance score σ′2 is compared with the gradient feature G. in Element-wise multiplication yields the processed gradient feature, i.e., σ′2×G. in The processed image features and the processed gradient features are added element-wise, resulting in σ′1×X′. in +σ′2×G in The result of element-wise summing of the processed image features and the processed gradient features is compared with the image feature X′. in Establishing a residual connection yields σ′1×X′. in +σ′2×G in +X′ in Thus, we obtain S′=(1+σ′1)×X in +σ′2×G in

[0163] Among them, S′ is the main fusion feature.

[0164] Step S207: Further perform a three-layer convolution operation on the main fused features.

[0165] The main fusion feature S′ is first normalized, then passed through a 3×3 convolution, a 1×1 convolution, a Leaky ReLU activation function, and another 1×1 convolution. It is then identity-connected to the main fusion feature S to obtain the output X′ of the second gradient-guided calibration unit 602. out To enrich the texture information in the image, the output X′ of the second gradient-guided calibration unit 602 is... out This corresponds to the enhanced image features in decoder layer 502. Enhanced image features in the decoder help restore image details and remove noise, resulting in a clearer and higher-quality image.

[0166] The training process of the low-light image enhancement algorithm model 30 is described below, including the steps shown in Figure 13.

[0167] Step S301: Select data pairs consisting of low-light image samples and infrared image samples from the dataset; select data pairs consisting of low-light image samples and normal-light image samples from the dataset.

[0168] Step S302: Generate the gradient map of the low-light image.

[0169] For example, the gradient map of a low-light image can be generated using the Sobel operator.

[0170] Step S303: Input the gradient map of the low-light image and the infrared image into the gradient reconstruction module 301; calculate the loss using the amplitude-enhanced gradient map of the low-light image and the gradient map of the normal-light image generated by the gradient reconstruction module 301.

[0171] The loss function is:

[0172] Among them, I gradient_enhance Gradient maps representing amplitude enhancement in low-light images; I gradient_gt This represents the gradient map calculated using the same Sobel operator for a normally lit image; H represents the image height; W represents the image width. It is a normalization factor that ensures the value of the loss function does not change with the image size; L gradient By measuring the average Euclidean distance between the gradient map representing the amplitude-enhanced, low-light image and the normal-light image, the gradient map of the amplitude-enhanced, low-light image is made as close as possible to the gradient of the normal-light image when minimizing the loss function, thereby increasing the image enhancement effect. In some embodiments, the Sobel operator is an edge detection operator for image processing that primarily detects edges in an image by calculating the gradients of the image in the horizontal and vertical directions. The Sobel operator is a gradient-based edge detection method that helps highlight edge regions in an image, making details in the image more prominent.

[0173] The gradient map of the amplitude-enhanced low-light image is used to extract gradient features G0 through a 3×3 convolution, and the gradient features G0 of the low-light image are also extracted through a 3×3 convolution. Gradient features G0 and G0 are then input into encoder module 302 and decoder module 303, respectively, and finally, a three-channel enhanced visible light image I is recovered through a single 3×3 convolution layer. enhance .

[0174] The overall loss function of the low-light image enhancement algorithm model 30 includes: L vgg =|φ(I enhance )-φ(I gt )||1, L total =L pixel +L vgg +L amphase +λ1L edge +λ2L gradient ,

[0175] in:

[0176] L pixel Pixel consistency loss is used to measure the enhancement of visible light image I. enhance and normal lighting image sample I gt Pixel-level differences between them ε is the normalization factor, H is the image height, W is the image width, 3 is the number of channels (the number of channels can be adjusted according to actual needs), and ε is a very small positive number to prevent division by zero.

[0177] L vgg Perceptual consistency loss is used to measure the enhancement of visible light image I. enhance The difference between the image sample and the image sample under normal lighting in the high-level feature space; φ(*) represents the output of certain layers of the pre-trained VGG-19 network. For example, the output of certain layers of the pre-trained VGG-19 network refers to the feature map extracted from a specific layer of the VGG-19 network; ||(*)||1 represents the L1 norm, which calculates the absolute difference between feature maps; the VGG-19 network is a deep convolutional neural network, widely used in image classification and feature extraction tasks. A specific layer can be a convolutional layer in the VGG-19 network. The output of the convolutional layer, as a feature map, contains high-level features of the image and can capture the structural and semantic information of the image;

[0178] L amphase Amplitude spectral consistency loss is used to measure the enhancement of visible light image I. enhanceThe difference in amplitude spectrum between the sample and the image under normal illumination in the frequency domain, where Amp(*) represents the amplitude spectrum obtained by Fourier transform; the remaining parameters are defined in the same way as the pixel consistency loss.

[0179] L edge Represents edge consistency loss, used to measure the enhancement of visible light image I. enhance The difference in edge features between the sample and the normally illuminated image. This represents the edge feature map obtained from the image using the Laplacian operator; the remaining parameters are defined in the same way as the pixel consistency loss.

[0180] L gradient This represents the gradient consistency loss, used to measure the enhancement of visible light images I. enhance The difference in gradient between L and normal illumination image samples gradient The formula is the same as the loss function formula in step S, the difference being that it is used to calculate the enhanced visible light image I. enhance The squared difference between the gradient of the image and the gradient of the normally illuminated image sample.

[0181] L total This represents the overall loss function, which comprehensively considers consistency at the pixel level, perceptual level, frequency domain, edge level, and gradient level to improve the image enhancement effect. λ1 and λ2 are used to adjust the weights of edge consistency loss and gradient consistency loss, and can be adjusted according to actual needs; for example, they can be set to 0.1 and 10 respectively.

[0182] In some embodiments of this application, the loss function may be selected from one or more parts as needed. In some embodiments of this application, where the primary focus is on pixel consistency of the image and the aim is to enhance the visible light image I... enhance When aiming to approximate normal lighting image samples at the pixel level, only pixel consistency loss L can be selected. pixel That is, L total =L pixel Similarly, in some embodiments of this application, only perceptual consistency loss, amplitude spectrum consistency loss, edge consistency loss, and gradient consistency loss can be selected. In some embodiments of this application, multiple loss functions can also be combined and selected according to requirements. The weights of the loss functions can be tuned experimentally. After calculating the overall loss, the low-light image enhancement algorithm model 30 is trained and updated using the Adam optimizer.

[0183] This application provides an air conditioner 100 in some embodiments, as shown in FIG15. The air conditioner 100 includes an indoor unit. The indoor unit includes an infrared imaging device and a camera device. The infrared imaging device can sample infrared images of the air-conditioned room, and the camera device can sample visible light images of the air-conditioned room. The visible light images include low-light images. In some embodiments, the air conditioner 100 further includes a processor 71, which includes a low-light image enhancement algorithm model 30, which includes: a gradient reconstruction module 301, which includes: a frequency domain processing branch 401, configured to convert the gradient map of the low-light image from the spatial domain to the frequency domain to extract the phase spectrum and amplitude spectrum of the low-light image; and a spatial domain processing branch 402, configured to fuse the gradient map of the low-light image with the infrared image in the channel dimension; the gradient reconstruction module 301 adds the residual of the spatial domain processing branch 402 to the amplitude spectrum output by the frequency domain processing branch 401 to obtain an enhanced amplitude spectrum; and obtains the amplitude-enhanced gradient map of the low-light image by inverse Fourier transform between the enhanced amplitude spectrum and the original phase spectrum. In some embodiments, the low-light image enhancement algorithm model 30 generates an enhanced visible light image through an encoder-decoder architecture based on the gradient features of the gradient map of the low-light image with amplitude enhancement and the image features of the low-light image. In some embodiments, the air conditioner 100 further includes a controller 72 configured to perform one or more of face recognition, gesture recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition based on the enhanced visible light image, and to control the operation of at least the indoor unit based on the recognition results. Controlling the operation of the indoor or outdoor unit based on the results of face recognition, gesture recognition, emotion recognition, gender recognition, age recognition, and vital sign recognition can employ algorithms from related technologies, which will not be elaborated here. In some embodiments, the processor 71 can also be located in the wired controller, central controller, or cloud server of the indoor unit.

[0184] In some embodiments, the processor 71 can incorporate a multi-level processing framework to divide low-light image enhancement into multiple stages. Each stage focuses on different image processing tasks, such as noise suppression, edge detection, and detail restoration. This ensures a more refined and diverse image processing process, thereby improving the quality and detail fidelity of the final image. To ensure rapid response in home appliances such as air conditioners, the processor 71 can be used in conjunction with dedicated hardware acceleration units (such as GPUs, TPUs, or FPGAs). This can significantly reduce image processing latency, especially in applications requiring real-time face recognition and gesture recognition, enhancing the real-time performance and smoothness of image processing and recognition. To further improve image processing, the processor 71 can enhance image quality through multi-channel data fusion (such as infrared, visible light, and depth images). For example, deep learning-based methods can automatically fuse the advantages of infrared and visible light images to improve image detail, particularly supplementing and restoring edge and texture information that is difficult to obtain in low-light environments. The controller 72 can integrate more sensor data, including environmental monitoring data (such as temperature, humidity, and air quality) in addition to image recognition, to comprehensively determine the operating status of the air conditioner. For example, when the controller 72 detects that a user's facial expression indicates "fatigue," it can automatically adjust the indoor temperature and humidity to create a more comfortable environment. The controller 72 can also make adjustments based on factors such as ambient noise and lighting conditions. It can also integrate an AI scene recognition system, allowing the controller 72 to adjust its control strategy not only based on static recognition results but also by dynamically sensing the environment of the air-conditioned room (such as a cinema, bedroom, or laboratory). For instance, during a movie screening, if the system detects that the room is in a low-light environment and the user is watching video content, the controller 72 can automatically reduce indoor lighting and adjust the air conditioner's fan speed and temperature to suit the user's viewing experience.

[0185] The air conditioner 100 provided in some embodiments of this application is particularly suitable for use in low light conditions, including not only different nighttime use environments, but also cinemas, recording studios, museums, laboratories, or hospital wards, so that the above-mentioned identification can be achieved and the corresponding control can be executed without adjusting the light.

[0186] The preceding embodiments primarily focus on enhancing low-light images. By combining frequency and spatial domain processing methods with a gradient reconstruction module, image quality in low-light environments is improved, resulting in enhanced visible light images. This process effectively preserves image details and structure, making it suitable for image processing needs in home appliances. The following embodiments further extend this image processing approach to multimodal image fusion. By combining the advantages of infrared and visible light images, a fusion recognition model is used to identify and analyze the emotions of target objects. By fusing thermal information from infrared images with detailed information from visible light images, not only is the accuracy and reliability of emotion recognition improved, but home appliances such as air conditioners can also better adapt to and regulate their environment, further enhancing the user experience.

[0187] Figure 16 illustrates the functional configuration of the image processing apparatus 1 disclosed in this application. In some embodiments, the image processing apparatus shown in Figure 1 and the image processing apparatus shown in Figure 16 may have the same structure and function.

[0188] In some embodiments of this application, the image processing apparatus 1 includes an acquisition unit 10 and a processing unit 20. In some embodiments of this application, the acquisition unit 10 may be integrated into the image processing apparatus 1. In some embodiments of this application, the acquisition unit 10 may also be connected to the image processing apparatus 1 via wired or wireless communication. In some embodiments of this application, the acquisition unit 10 is configured to acquire an image of a target scene. In some embodiments, the acquisition unit 10 is also referred to as an acquisition unit. The image includes infrared images and visible light images, which are sampled according to a preset time sequence. In some embodiments of this application, the data sampling of the infrared images and visible light images is implemented by an infrared imaging device and a camera device, respectively.

[0189] Infrared imaging devices are equipment that use infrared radiation to create images. All objects emit infrared radiation of different wavelengths depending on their temperature; the higher the temperature, the greater the radiation intensity and the shorter the wavelength. Infrared imaging devices detect infrared wavelength signals and convert them into different electrical signals to capture thermal energy, which is then further converted into images (or videos) for observation and analysis. The images output by infrared imaging devices are usually called infrared images or thermal images, where different colors represent different temperatures. For example, red or white can represent heat sources, while blue represents cooler areas. Camera devices, on the other hand, are equipment that captures visible light wavelengths to create images. They can convert light signals into image signals to record the corresponding scene. Camera devices are imaging devices based on related technologies.

[0190] In some embodiments of this application, the infrared imaging device and the camera device cover the same target scene from their respective perspectives. The perspectives of the infrared imaging device and the camera device can be fixed or adjustable. In some embodiments of this application, the infrared imaging device and the camera device sample according to a preset time sequence. The infrared imaging device outputs an infrared image according to the preset time sequence, and the camera device outputs a visible light image according to the preset time sequence. Both the infrared image and the visible light image are time-series image data, consisting of multiple consecutive image frames, with each time step (frame) having a corresponding infrared image and a visible light image.

[0191] As shown in Figure 17, in some embodiments of this application, the image processing apparatus further includes a storage unit 40. The storage unit 40 includes, for example, a non-volatile memory for storing a database of infrared images output according to a preset time series and visible light images output according to a preset time series. The storage unit 40 can be a built-in solid-state memory or a removable memory card, a card recording and playback unit for recording or reproducing access to the memory card, a hard disk, an optical disk, etc. In some embodiments, the storage unit 40 is also referred to as a storage unit.

[0192] As shown in Figure 18, the processing unit includes a central processing unit (CPU) 201, a read-only memory (ROM) 202, and a random access memory (RAM) 203. The CPU 201 executes various processing tasks based on programs stored in the ROM 202 or programs loaded from storage units into the RAM 203. Furthermore, the RAM 203 appropriately stores data required by the CPU 201 to perform these tasks. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 205. An input / output interface 204 is also connected to the bus 205.

[0193] As shown in Figure 19, the processing unit 20 is equipped with a fusion recognition model 211. The fusion recognition model 211 is configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network to identify the emotion of a target object in the image and / or identify the emotion changes of the target object in the image within a set time period. In some embodiments, the processing unit 20 is also referred to as a processing unit.

[0194] In some embodiments, the fusion recognition model 211 uses an infrared feature extraction network and a visible light feature extraction network to extract feature vectors from infrared and visible light images, respectively. The infrared and visible light image feature vectors are then fused, and the fused feature vector is used for emotion recognition. For example, a trained classifier can be used to identify the emotion of a target object (a person in a target scene). The fusion recognition model 211 also uses the fused feature vector to identify the emotion changes of the target object over a set time period, for example, using an RNN or LSTM model. An example architecture of the fusion recognition model 211 will be described in detail below. Because infrared images can capture thermal information of the target object under low-light conditions, while visible light images provide rich detail information under strong light conditions, the accuracy and reliability of emotion recognition are improved.

[0195] In some embodiments, RNN (Recurrent Neural Network) and LSTM (Long Short-Term Memory) are two deep learning models primarily used for processing sequential data. They are suitable for tasks such as time series prediction, natural language processing, and speech recognition because they can handle the temporal dependencies between input data. In some embodiments, the fusion recognition model 211 not only fuses features from infrared and visible light images but also performs data fusion at different levels. Preliminary feature fusion can be performed at the output layer of the feature extraction network, concatenating or integrating the feature vectors of infrared and visible light images through weighted fusion to form a fused feature vector. Furthermore, fusion can be performed in deeper layers of the network, dynamically selecting and weighting features from different sources through an attention mechanism to achieve context-based multimodal fusion, further improving the accuracy of emotion recognition. In recognizing changes in the emotion of the target object, the fusion recognition model 211 can not only utilize LSTM or RNN to process time series data but also introduce an emotion context analysis module to capture temporal patterns related to emotion. For example, based on the time-series characteristics of emotional changes, the ability to model emotional changes can be further enhanced by designing a spatiotemporal convolutional neural network (ST-CNN) to capture the spatiotemporal correlation of emotional fluctuations. Furthermore, adding an emotional state transition model (such as a Markov chain-based model) to predict possible emotional state transitions also helps improve the predictive ability of emotional change identification.

[0196] As shown in Figure 20, in some embodiments of this application, the processing unit 20 further includes an emotion classification model 212. The emotion classification model 212 is configured to identify the emotion of the target object in the image within a set time period based on the infrared image and / or visible light image after the fusion recognition model 211 identifies the emotion changes of the target object within the set time period, providing preliminary emotion analysis results and quickly filtering out target objects that need further analysis. Then, the emotion classification model 212 performs more accurate emotion identification of the target image based on the infrared image and / or visible light image. Through a staged processing approach, the efficiency of emotion recognition is effectively improved, and computing resources are saved. The emotion classification model 212 can employ rule-based models, machine learning-based models, deep learning-based models, etc. An example architecture of the emotion classification model 212 will be described below.

[0197] As shown in Figure 21, in some embodiments of this application, the processing unit 20 further includes an emotion segmentation model 213. The emotion segmentation model 213 is configured to generate emotion morphology segmentation results of the target object in the infrared and / or visible light images based on the infrared and / or visible light images after the fusion recognition model 211 identifies the emotion changes of the target object in the image within a set time period. The fusion recognition model 211 first identifies the emotion changes of the target object within the set time period, providing preliminary emotion analysis results, quickly filtering out target objects that need further analysis, and providing the user's overall emotional tendency; for example, positive, negative, or neutral. Next, the emotion segmentation model 213 further provides fine-grained analysis, thereby providing a multi-level understanding of the overall emotion. The emotion segmentation model 213 can employ rule-based models, machine learning-based models, deep learning-based models, etc. An example architecture of the emotion classification model 212 will be described below.

[0198] As shown in Figure 22, in some embodiments of this application, the processing unit 20 includes an emotion classification model 212 and an emotion segmentation model 213. The emotion classification model 212 is configured to identify the emotion of the target object in the image within a set time period based on infrared and / or visible light images after the fusion recognition model 211 identifies the emotion changes of the target object in the image within a set time period. The emotion segmentation model 213 is configured to generate an emotion morphology segmentation result of the target object in the infrared and / or visible light images based on infrared and / or visible light images after the fusion recognition model 211 identifies the emotion changes of the target object in the image within a set time period. The fusion recognition model 211 first identifies the emotion changes of the target object within a set time period, providing preliminary emotion analysis results and quickly filtering out target objects that need further analysis. Then, the emotion classification model 212 performs more accurate emotion identification of the target image based on infrared and / or visible light images. Through a staged processing approach, the efficiency of emotion recognition is effectively improved, saving computing resources. Simultaneously, the emotion segmentation model 213 further provides fine-grained analysis, thereby providing a multi-level understanding of the overall emotion.

[0199] In some embodiments of this application, the emotion classification model 212 can integrate adaptive learning capabilities to update the model in real time and adapt to differences in emotional expression among different users or scenarios. For example, by continuously learning from the user's emotional changes, the classification accuracy can be optimized. By fusing data from different modalities (infrared and visible light), the emotion classification model can achieve more robust emotion recognition results. Based on multimodal learning, the model can better cope with low-light or unclear image environments. Emotion classification not only relies on the image itself but can also integrate other contextual information (such as time, location, environmental conditions, etc.) to enhance the accuracy of emotion recognition. Combining local region emotion analysis, the emotion segmentation model 213 can more accurately identify the contribution of each part of the facial expression to the overall emotion, thereby providing more detailed emotion analysis. Combining infrared and visible light images, the emotion segmentation model 213 can analyze emotions from different dimensions, such as combining temperature changes in infrared images with facial expressions in visible light images, to more comprehensively understand the emotional state of the target object. The emotion segmentation model 213 can track the emotional changes of the target object by combining time series data, generating an emotion change trend map based on long-term change patterns to help provide more accurate emotion assessment.

[0200] In some embodiments of this application, the image processing apparatus performs a refined emotion recognition method based on the fusion of infrared and natural images, as shown in FIG23. In some embodiments of this application, as shown in FIG23, the refined emotion recognition method based on the fusion of infrared and natural images consists of three stages: data acquisition and preprocessing, feature extraction and fusion, and emotion recognition. The three stages are described below.

[0201] Phase 1: Data Acquisition and Preprocessing

[0202] As shown in Figure 24, step S101: Data sampling. For detailed steps of data sampling, please refer to the description above; they will not be repeated here.

[0203] Step S102: Preliminary data screening.

[0204] Infrared images and visible light images, both output according to a preset time series, are stored in a database. Preliminary filtering or screening of the multi-source image data (including infrared and visible light images) removes irrelevant, low-quality, or invalid data for subsequent analysis and processing. This preliminary filtering or screening includes, but is not limited to, infrared image quality detection and visible light image quality detection. In some embodiments of this application, blurred, low-resolution, unevenly illuminated images, and images damaged due to transmission problems are removed during infrared and visible light image quality detection. Infrared and visible light image quality detection can employ relevant techniques in the field of image processing, which will not be elaborated upon here.

[0205] Step S103: Data Normalization

[0206] Infrared and visible light images are normalized to unify images of different modalities (including infrared and visible light images) to the same scale range.

[0207] Step S104: Create abnormal data

[0208] In databases, normal data constitutes the vast majority. Creating anomalous data can enrich the database and improve the performance of emotion recognition models when processing anomalous data. In some embodiments of this application, "anomalous data" refers to image data that does not conform to or cannot represent a true emotional state. In some embodiments, the process of creating anomalous data may include: selecting image data whose facial temperature changes exceed a preset facial temperature change threshold as "anomalous data". Human facial temperature is affected by various factors, and emotional reactions can cause changes in facial blood flow, thereby altering skin temperature. Under normal circumstances, changes in skin temperature are gradual. When facial temperature changes exceed a preset facial temperature change threshold, the facial temperature changes become extreme and unconventional, and can be considered anomalous data. In some embodiments, the process of creating anomalous data may include: selecting image data whose emotional changes exceed a preset emotional change threshold as "anomalous data". Emotional changes exceeding a preset emotional change threshold represent emotional fluctuations that are out of the ordinary. For example, if an expression undergoes a large change in a short period of time (changing from a smile to anger in a very short time; "smile," "anger," etc., can be represented by corresponding numbers), such excessive emotional fluctuations do not conform to common emotional expression patterns, and this type of data is considered anomalous data.

[0209] Step S105: Data Labeling

[0210] Since model training and testing require both normal and abnormal data, "normal" / "abnormal" labels are created. For infrared images, if facial temperature is abnormal within a set time period (e.g., facial temperature change exceeds a preset facial temperature change threshold), the infrared image for that time period is labeled as abnormal, for example, represented by the number "1"; if facial temperature is normal within the set time period (e.g., facial temperature change does not exceed the preset facial temperature change threshold), the infrared image for that time period is labeled as normal, for example, represented by the number "0". For visible light images, if emotional changes are abnormal within a set time period (e.g., emotional changes exceed a preset emotional change threshold), the visible light image for that time period is labeled as abnormal, for example, represented by the number "1"; if emotional changes are normal within the set time period (e.g., emotional changes do not exceed the preset emotional change threshold), the visible light image for that time period is labeled as normal, for example, represented by the number "0".

[0211] Through the above steps, the first stage completes the acquisition and processing of the multi-source image database, ensuring data quality and reliability.

[0212] In some embodiments of this application, data annotation can also indicate whether there are people in the scene, for example, "1" for someone and "0" for no one. The annotated multi-source image database can be used for model training. In some embodiments of this application, both infrared images and visible light images can have an emotion category label. The emotion category label can be an integer representing different emotion categories, such as: 0: happy; 1: sad; 2: angry; 3: surprised; 4: fear; 5: disgust; 6: expressionless. The emotion category label can also be converted into a vector of length equal to the number of categories through one-hot encoding, for example, happy is [1,0,0,0,0,0,0].

[0213] In some embodiments of this application, pixels in infrared and visible light images are labeled with a certain emotion category. For example, the mouth area in a facial image might be labeled "happy," while other background areas might be labeled "expressionless." Infrared and visible light images can be labeled using an emotion tag map, which can be an image of the same size as the infrared and visible light images, where the value of each pixel represents the emotion category at that location. In some embodiments of this application, the infrared and visible light images also include markers indicating the timestamps of emotion changes.

[0214] Phase 2: Feature extraction and fusion.

[0215] In some embodiments of this application, feature extraction is implemented by a feature extraction network. In some embodiments of this application, the feature extraction network includes an infrared feature extraction network 214 and a visible light feature extraction network 215. As shown in Figure 25, in some embodiments of this application, the infrared feature extraction network 214 includes a four-layer Transformer encoder to ultimately output an infrared image feature vector. Feature extraction of infrared images not only needs to consider the texture and structural information of the image, but also needs to identify features such as temperature distribution, the location and changes of heat sources. The Transformer encoder is one of the neural network architectures used in natural language processing and other deep learning tasks.

[0216] In some embodiments of this application, each layer of the Transformer encoder in the infrared feature extraction network 214 may include: a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections. The multi-head self-attention mechanism can capture the dependencies between different parts of the input infrared image data. For example, infrared images contain unevenly distributed thermal information, and the self-attention mechanism can effectively capture such long-range dependencies to understand the global distribution of heat sources. The feedforward neural network can further process the results of the self-attention output, while layer normalization and residual connections ensure the stability of the training process.

[0217] In some embodiments of this application, the lowest-level Transformer encoder receives the input infrared image I. IR The input infrared image is transformed into a preliminary feature vector, and a self-attention mechanism is used to capture local relationships, learning preliminary features (such as hotspots and temperature distribution). The middle two Transformer encoder layers are used to further learn complex patterns and global information in the input infrared image. For example, hotspots in the image are local features in the preliminary feature extraction stage. These local features are correlated in the middle two Transformer encoder layers to learn more representative global features. The highest Transformer encoder layer integrates features from different levels, finally obtaining a high-dimensional feature representation containing global image information, i.e., the infrared image feature vector Z. IR .

[0218] In some embodiments of this application, the infrared feature extraction network 214 further includes a sigmoid function, which is applied to the feature vector output by the highest-level Transformer encoder to output an infrared image encoded feature vector. In some embodiments of this application, the visible light feature extraction network 215 includes a four-layer Transformer encoder to ultimately output a visible light image feature vector Z. VIS In some embodiments of this application, the visible light feature extraction network 215 includes a four-layer Transformer encoder. The structure of the four-layer Transformer encoder is the same as that of the infrared feature extraction network 214, and will not be described again here. In some embodiments of this application, the visible light feature extraction network 215 also includes a sigmoid function, which applies the sigmoid function to the feature vector output by the highest-layer Transformer encoder to output a visible light image encoded feature vector.

[0219] In some embodiments of this application, the infrared feature extraction network 214 includes an Xception network, a residual module, and a depthwise separable convolution (these three parts together form an improved Xception network). The Xception network itself is based on a convolutional neural network architecture, dividing the convolution operation into two steps: first, processing each channel with depthwise convolution, and then processing the combination between different channels with pointwise convolution (1x1 convolution). Furthermore, the residual module and depthwise separable convolution extract high-dimensional features from infrared and visible light images, enhancing the ability to extract emotional feature information and generating high-dimensional infrared image feature maps. In addition, the infrared image feature network can also combine residual connections and layer normalization to mitigate gradient vanishing and accelerate model training.

[0220] In some embodiments of this application, the visible light feature extraction network 215 also includes an Xception network, a residual module, and a depthwise separable convolution to generate a high-dimensional visible light image feature map. The high-dimensional infrared image feature map and the visible light image feature map can be further transformed into an infrared image feature vector Z through global pooling. IR and visible light image feature vector Z VIS .

[0221] In some embodiments of this application, the infrared feature extraction network 214 and the visible light feature extraction network 215 can be fused through multimodal learning. Infrared images provide temperature information related to physiological states, while visible light images provide visual information related to emotions. By fusing information from these two different modalities through a multi-layered Transformer encoder network, the model can perform more comprehensive emotion recognition of the target object. For example, infrared images can capture temperature changes caused by emotional changes, while visible light images further reveal emotional details through facial expressions and body language. The multimodal network can integrate information from both to provide more stable and accurate predictions of emotional changes. In poor lighting conditions, the thermal information from infrared images can supplement the deficiencies of visible light images, while visible light images can provide detailed information lacking in infrared images. This complementarity can improve the adaptability and reliability of the emotion recognition system in various environments.

[0222] In some embodiments of this application, an example architecture of an infrared feature extraction network or a visible light image extraction network is shown in Figure 26, including an input layer, as shown in Figure 27; an intermediate layer, as shown in Figure 28; and an output layer. For example, the input image can be an infrared image or a visible light image with a width and height of 299 pixels and 3 color channels (RGB).

[0223] In the input layer, an initial convolution operation is first performed. The initial convolution uses a 3x3 convolution kernel, has 32 output channels, a stride of 2x2, and uses ReLU activation function. The initial convolution is used to perform preliminary feature extraction on the input image and reduce the spatial size of the feature map.

[0224] After the initial convolution operation, the input layer includes three consecutive convolutional modules, each containing two paths. In the first convolutional module, the first path uses a 1x1 kernel with a stride of 2x2, while the second path performs depthwise separable convolution (128 channels, 3x3 stride), ReLU activation, and max pooling (3x3 pixels, 2x2 stride). The outputs of the first and second paths are superimposed to combine convolution and depthwise separable convolution, improving feature extraction capability and computational efficiency. In the second convolutional module, the third path uses a 1x1 kernel with a stride of 2x2, while the fourth path performs depthwise separable convolution (256 channels, 3x3), ReLU activation, and max pooling (3x3 pixels, 2x2 stride). The outputs of the third and fourth paths are superimposed to further extract features, and the original features are preserved through residual connections. In the third convolution module, the fifth path uses a 1x1 convolution kernel with a stride of 2x2, while the second path performs depthwise separable convolution (728 channels, stride 3x3), ReLU activation, and max pooling (3x3 pixels, stride 2x2). The outputs of the first and second paths are superimposed to continue extracting features, and the original features are preserved through residual connections.

[0225] The intermediate layers employ ReLU activation and depthwise separable convolutions (728 channels, 3x3 stride), and are then stacked with the intermediate layer input to further extract and represent features while maintaining computational efficiency. This process is repeated multiple times, for example, eight times.

[0226] The output layer employs ReLU activation, followed by depthwise separable convolution (728 channels, 3x3 stride), max pooling (3x3 pixels, 2x2 stride), and then superimposed with the result of a 1x1 convolution kernel with a 2x2 stride. This superimposed result is then subjected to another depthwise separable convolution (1536 channels, 3x3 stride), ReLU activation, another depthwise separable convolution (2048 channels, 3x3 stride), and global average pooling. Through multiple depthwise separable convolutions, features are further extracted, and global average pooling transforms the feature map into a fixed-size feature vector, namely the infrared image feature vector Z. IR and visible light image feature vector Z VIS .

[0227] An infrared and visible light feature extraction network based on an improved Xception network is constructed. Initial convolutions perform preliminary feature extraction on the input image, reducing the spatial size of the feature map. In multi-path convolutions, combining traditional convolutions and depthwise separable convolutions improves feature extraction capability and computational efficiency. Residual connections preserve the original features, enhancing the network's training efficiency and expressive power. Multiple repetitions of depthwise separable convolutions and residual connections further extract and represent features while maintaining computational efficiency. Multiple depthwise separable convolutions and max pooling further extract features, and global average pooling transforms the feature map into a fixed-size feature vector.

[0228] In some embodiments of this application, the infrared images I input to the infrared feature extraction network can be... IR And the visible light image input to the visible light image feature network with added location information I VIS More specifically, it has temporal location information. The infrared image I... IR and visible light image I VIS Alignment is performed by timestamp. Assuming a set of infrared and visible light images are acquired within each time period, these images are arranged chronologically. The aligned infrared and visible light images are then grouped into a matrix. Assuming N infrared and N visible light images are acquired within each time period, these images can be arranged into a matrix chronologically. Position encoding is then applied to the infrared images within each time period. The position encoding formula is as follows:

[0229] Where pos is an integer representing the number of infrared images within a time period. For example, if 10 infrared images are collected within a time period, then pos∈[0,9]; PE represents the position code of each infrared image; i represents the data index, which is calculated using a sine function in even positions and a cosine function in odd positions; when i=0, 2i and 2i+1 represent the first and second infrared images within a certain time period, respectively; d is the number of infrared images within each time period.

[0230] The visible light image for each time period is position-coded using the following formula:

[0231] Where pos is an integer representing the number of visible light images within a time period. For example, if 10 visible light images are collected within a time period, then pos∈[0,9]; PE represents the position code of each visible light image; i represents the data index, which is calculated using a sine function in even positions and a cosine function in odd positions; when i=0, 2i and 2i+1 represent the first and second visible light images within a certain time period, respectively; d is the number of visible light images within each time period.

[0232] In some embodiments of this application, each infrared image is added to a calculated position code and then input into an infrared feature extraction network. In some embodiments of this application, each visible light image is added to a calculated position code and then input into a visible light feature extraction network. Through these methods, the position code retains the temporal information of the input data, enabling the feature extraction network to understand the temporal positional relationships of the data. In some embodiments of this application, a probability vector C can be further generated based on the infrared image feature vector and the visible light image feature vector. predict In some embodiments of this application, the infrared image feature vector and the visible light image feature vector are concatenated along the feature dimension to obtain a higher-dimensional feature vector. In some embodiments of this application, the infrared image feature vector and the visible light image feature vector are weighted and summed to obtain a fused feature representation Z. fused Z is represented by the fused features. fused For example, the fused feature Z fused The feature is passed to a fully connected layer for linear transformation, and the feature after linear transformation is denoted as Z. mapped The result of the linear transformation is processed using the sigmoid function to obtain the probability vector C. predict In some embodiments of this application, after obtaining the linearly transformed feature Z... mapped Then, we further added attention calculation steps. The formula for the attention mechanism is as follows:

[0233] Q, K, and V are the inputs to the attention mechanism, representing the query, key, and value matrices, respectively. k This represents the dimension of the attention mechanism; Attention(Q,K,V) represents the attention output, and K... T This indicates that K is transposed, and softmax represents normalizing the attention weights. Specifically, three different linear layers are used for Z. mapped Perform the transformation to generate Q, K, and V, where Q = Z mapped ·W Q K = Z mapped ·W K V = Zmapped ·W V W Q W K and W V Given a trainable weight matrix; compute the dot product of the query and the key, and divide by . Scaling is performed, and the softmax function is applied to the score matrix to obtain the attention weights. The attention output Z is obtained by weighted summation of the value matrix using attention weights. attention , The attention output is processed by the sigmoid function to obtain the probability vector C. predict C predict = sigmoid(Z attention ).

[0234] In some embodiments of this application, the probability vector C can be generated. predict During the process, relevant data from indoor sensors are integrated. The fusion recognition model is configured to integrate infrared image feature vectors extracted by an infrared feature extraction network, visible light image feature vectors extracted by a visible light feature extraction network, and detection value feature vectors extracted from sensor detection data to identify the emotion of a target object in the image and / or to identify the emotional changes of a target object in the image within a set time period. That is, the infrared image feature vectors and visible light image feature vectors are mapped to a space related to the type of indoor sensor data. The detection value feature vectors extracted from sensor detection data can be obtained using time-domain feature extraction methods in related technologies, such as mean, standard deviation, etc., or they can be obtained using time-frequency domain feature extraction methods.

[0235] In some embodiments of this application, the infrared image feature vector, the visible light image feature vector, and the detection value feature vector are concatenated along the feature dimension to obtain a higher-dimensional feature vector. In some embodiments of this application, the infrared image feature vector, the visible light image feature vector, and the detection value feature vector are weighted to obtain a fused feature representation Z. fused Furthermore, through a fully connected layer, Z... fused This is mapped to a 1×n vector, where n represents the number of sensor types. For example, assuming three sensor data points (e.g., temperature, humidity, and heart rate), then n = 3. Sensor detection values ​​can further enhance the model's recognition results. Then, the above steps are performed to obtain the probability vector C. predict Fusing the probability vector of the feature vectors of detected values ​​can improve the accuracy of emotion recognition. During the training of the feature extraction network, this is achieved by minimizing the probability vector C. predict With real label C real The distance between them is used to optimize network parameters. Loss function. indoorThis can be represented as: loss indoor =(C predict -C real ) 2

[0236] During training, the forward and backward propagation processes are repeated until the model converges or reaches the predetermined number of training rounds.

[0237] In some embodiments of this application, when a four-layer Transformer encoder is selected as the infrared feature extraction network and the visible light infrared feature extraction network, the probability vector C can be further... predict After fusing with low-level image features (such as the output of convolutional layers), the data is input into an improved Xception network to further extract sentiment change signals from time-series data. Probability vector C predict Low-level image features can be achieved through simple connection operations or weighted fusion. The fused features not only contain high-level semantic information of emotion but also combine low-level visual features of the image. Improving the Xception network helps to learn the dynamic changes between images and emotions at the temporal level, capturing the detailed features behind emotional changes, especially subtle changes that may not be fully extracted from image features.

[0238] As shown in Figure 29, the improved Xception network consists of an input layer, multiple residual blocks, and an output layer. In the input layer, initial feature extraction and normalization are performed first through a Conv2D layer and a BatchNorm layer, followed by further feature extraction through another Conv2D layer and a BatchNorm layer. Each residual block comprises two branches. For the first branch, depthwise separable convolution and normalization are performed first through a Sep-Conv2D layer and a BatchNorm layer; then, further feature extraction is performed through another Sep-Conv2D layer and a BatchNorm layer; finally, feature fusion is performed through a Conv2D layer. For the second branch, convolution and normalization are performed through a Conv2D layer and a BatchNorm layer. The outputs of the two branches are summed to obtain the output of the residual block. The output layer performs final feature extraction through a Conv2D layer; then, the feature map is compressed into a vector through a Global Avg Pooling 2D layer; finally, a Softmax layer outputs the classification probability. The improved Xception network combines depthwise separable convolutions and batch normalization, which can improve computational efficiency while maintaining high performance.

[0239] Using the aforementioned fusion recognition model, it is possible to identify whether a person experiences emotional changes within a set time period based on the fusion result of the input infrared and visible light images. In some embodiments, the fusion recognition model can also generate a classification of a person's emotions.

[0240] In some embodiments, the fusion recognition model is configured to perform the method shown in FIG30 to fuse the infrared image feature vector extracted by the infrared feature extraction network and the visible light image feature vector extracted by the visible light feature extraction network:

[0241] Step S201: Concatenate or weight the feature vectors of the infrared image and the visible light image to obtain the fused features.

[0242] Step S202: Apply the attention mechanism based on the fused features to obtain the attention output.

[0243] Step S203: Process the attention output using the sigmoid function to obtain the probability vector;

[0244] Step S204: Identify the emotion of the target object in the image based on probability vectors and / or identify the emotional changes of the target object in the image within a set time period.

[0245] In some embodiments of this application, the emotion classification model is a combination of a multi-layer convolutional neural network (CNN) and a Swin Transformer network. The image is sequentially passed through the CNN and the Swin Transformer network to extract encoded features. Emotion classification probabilities are generated based on these encoded features, and the emotion of the target object in the image is identified based on these probabilities. In some embodiments, the emotion classification model is used to encode feature Z. p The extraction process involves, for example, a four-layer convolutional neural network (CNN). The CNN is responsible for extracting local features, and the convolutional layers use a sliding window operation to locally perceive the input data, thereby capturing the local structural information of the data. The Swin Transformer is responsible for extracting global features by dividing the image or sequence into multiple windows and performing a self-attention mechanism within each window, effectively extracting global, abstract semantic information. In some embodiments of this application, the input data first enters the CNN to obtain local features Z. local Then the local feature Z local Then, the global feature Z is obtained through the Swing Transformer network. global Global feature Z global After passing through two more CNN layers, the features undergo certain non-linear transformations to extract the encoded feature Z. p To improve the model's ability to recognize emotions, a 1×1 emotion classification probability B is finally generated using the Sigmoid function.c , representing the probability that the emotion classification model predicts for the emotion category. The emotion classification model outputs the emotion category probability. The loss function used during the training of the emotion classification model is: loss p_c = (B c -B real ) 2

[0246] Among them, B real The emotion category label is used for annotation (e.g., when the emotion is "anger"), B real =1; When the emotion is "expressionless", B real =0, depending on the setting of the emotion category label). The Swin Transformer (Shifted Window Transformer) network is a visual Transformer architecture designed to solve the efficiency problems encountered by Transformers in computer vision tasks. Its design balances the locality and globality of images, overcoming the high computational complexity of self-attention mechanisms while retaining the modeling capabilities of Transformers. The Swin Transformer has demonstrated good performance in multiple vision tasks, such as image classification, object detection, and semantic segmentation.

[0247] In some embodiments of this application, if the fusion recognition model identifies a change in a person's emotion within a certain time period, the image from the last moment is selected and input into the emotion classification model for recognition and judgment. The last image can more accurately reflect the emotional state and distribution of people within a certain time period. In some embodiments of this application, the emotion segmentation model is a combination of a multi-layer convolutional neural network (CNN) and a Swin Transformer network. The image is sequentially passed through the CNN and the Swin Transformer network to extract encoded features. The encoded features are input into the network combination of the transposed CNN and the Swin Transformer network for decoding, generating the emotion morphology segmentation result of the target object in the image. The emotion segmentation model is used to encode feature Z. p The extraction process involves, for example, a four-layer convolutional neural network (CNN). The CNN is responsible for extracting local features, and the convolutional layers use a sliding window operation to locally perceive the input data, thereby capturing the local structural information of the data. The Swin Transformer is responsible for extracting global features by dividing the image or sequence into multiple windows and performing a self-attention mechanism within each window, effectively extracting global, abstract semantic information. In some embodiments of this application, the input data first enters the CNN to obtain local features Z. local Then the local feature Z localThen, the global feature Z is obtained through the Swing Transformer network. global Global feature Z global After passing through two more CNN layers, a certain nonlinear transformation is performed to extract the encoded feature Z. p This improves the model's ability to recognize emotions. Feature Z p The input is fed into a four-layer network combination of a transposed convolutional neural network and a Swing Transformer network for decoding, and the decoded result is the emotion morphology segmentation result X. decode Compare the emotion label chart X real And the emotional pattern segmentation result X decode We use both cross-entropy loss (BCEWithLogitsLoss) and Dice loss to determine X. real With X decode The degree of similarity between them. Assume the emotion label map X... real The regions labeled with emotion are assigned a value of 1, while the remaining background areas are assigned a value of 0 (morphological labels). Cross-entropy loss is used to evaluate the segmentation result X. decode The probability values ​​are converted to values ​​between 0 and 1 using a Sigmoid function. Then, the binary cross-entropy loss is calculated between these probability values ​​and the morphological labels of the converted sentiment label map. The formula for the binary cross-entropy loss is: BCE = -[y·log(t) + (1-y)·log(1-t)]

[0248] Where y represents the transformed morphological label, and t represents the probability value between 0 and 1 of the segmentation result after being transformed by the Sigmoid function.

[0249] Dice loss measures the degree of overlap between two images and is a commonly used loss function in image segmentation. A smaller value indicates a higher degree of overlap and a better segmentation result. The formula for calculating Dice loss is:

[0250] The loss function of the emotion morphology segmentation network seg The sum of the Dice loss and the BCEWithLogitsLoss loss is expressed as: loss seg = 0.5 × BCE + 0.5 × Dice

[0251] In some embodiments of this application, if the fusion recognition model identifies a change in a person's emotions within a certain time period, the image from the last moment is selected and input into the emotion segmentation model for recognition and judgment. The last image can more accurately reflect the emotional state of a person within a certain time period. Assuming that the data from the first five minutes is input into the fusion recognition model, the fusion recognition model can determine that the person has experienced an emotional change during this period. The last indoor image from these five minutes is selected as the final emotional state within this period. This image is input into a trained emotion classification network. The output of the emotion classification network helps to determine the emotional state, achieving real-time feedback and intervention as much as possible. The emotion classification model is used to identify the user's basic emotional state (such as happiness, sadness, anger, etc.) and quickly classify the user's emotions. At the same time, this image is input into the trained emotion segmentation model, which uses the emotion segmentation model to perform a detailed analysis of the emotions, such as distinguishing between a "satisfied smile" and a "polite smile".

[0252] Some embodiments of this application provide an air conditioner. The air conditioner includes an indoor unit, which includes an infrared imaging device, a camera device, and an indoor fan. The infrared imaging device can sample and store infrared images of the air-conditioned room according to a preset time sequence. The camera device can sample and store visible light images of the air-conditioned room according to a preset time sequence. The indoor fan is configured to deliver air to the air-conditioned room. In some embodiments, the indoor unit further includes a processing unit and a control unit. The processing unit includes a fusion recognition model, which is configured to fuse infrared image feature vectors extracted by an infrared feature extraction network and visible light image feature vectors extracted by a visible light feature extraction network to identify the emotion of a target object in the image and / or identify the emotion changes of the target object in the image within a set time period. In some embodiments, the control unit is also referred to as a control unit. The control unit is configured to adjust the operation of the indoor fan and / or adjust the set temperature of the air-conditioned room based on the identified emotion of the target object and / or the identified emotion changes of the target object within a set time period.

[0253] In some embodiments of this application, in addition to infrared and visible light image feature extraction, the processing unit can also integrate data from multiple sensors, such as temperature and humidity sensors and sound sensors, to further enhance the accuracy of emotion recognition. Infrared imaging devices can provide valuable information about human body temperature distribution, heat source location, etc., while visible light images provide detailed data on facial expressions, postures, and movements for emotion recognition. Combining sensor data, the processing unit can not only capture emotional features in environments with insufficient visible light, but also more comprehensively analyze the user's physiological state and psychological reactions. The processing unit can use online learning methods to gradually train and improve based on the user's historical emotional data. For example, the processing unit can adjust the emotion recognition model and adjustment strategy based on the user's long-term emotional feedback and air conditioning usage habits. This can not only improve the accuracy of personalized adjustment, but also enhance the interactive experience between the air conditioner and the user, enabling the air conditioner to proactively adapt to the needs of different users. For example, after recognizing changes in the user's emotions, it can automatically adjust the environmental settings to improve comfort. The control unit can not only adjust the fan speed and temperature based on emotion recognition, but also adjust them according to different situational needs. For example, when the system detects that a user is under significant stress or experiencing anxiety, the control unit can activate an air freshening mode or a soothing mode. By altering airflow patterns and temperature / humidity settings, it can help alleviate the user's emotional state. When a user exhibits fatigue or drowsiness, the air conditioner can improve user comfort and keep them alert by adjusting the temperature and increasing negative ion release. The control unit can also collaborate with a cloud platform, leveraging the large datasets accumulated there for more accurate and personalized control. The cloud platform can analyze emotional data from multiple user groups, providing group behavior pattern analysis to help the air conditioning system predict and adapt to specific needs in various situations. This enables cross-scenario control, allowing the control unit to provide tailored environmental adjustment strategies based on the specific context when a user uses the air conditioner in different locations or scenarios.

[0254] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples. The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing apparatus, comprising: an acquisition unit configured to acquire a low-light image and an infrared image of a target scene; a processing unit comprising: a low-light image enhancement algorithm model comprising: a gradient reconstruction module configured to obtain a gradient map of an amplitude-enhanced low-light image; 2. The image processing apparatus according to claim 1, wherein: the low-light image enhancement algorithm model generates an enhanced visible light image based on gradient features of the gradient map of the amplitude-enhanced low-light image and image features of the low-light image through an encoder-decoder architecture. the gradient reconstruction module comprises: a frequency domain processing branch configured to convert the gradient map of the low-light image from a spatial domain to a frequency domain to extract a phase spectrum and an amplitude spectrum of the low-light image; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in a channel dimension; the gradient reconstruction module adds a residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains the gradient map of the amplitude-enhanced low-light image through inverse Fourier transform between the enhanced amplitude spectrum and an original phase spectrum.

3. The image processing apparatus according to claim 1 or 2, wherein: the low-light image enhancement algorithm model comprises: an encoder module comprising: a plurality of encoder layers; the encoder layers comprise: a first gradient-guided calibration unit configured to: extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced low-light image and image features of the low-light image; generate processed image features based on the image channel importance scores and the image features; generate processed gradient features based on the gradient channel importance scores and the gradient features; generate main fusion features based on the processed image features and the processed gradient features, and a residual connection with the image features; 4. The image processing apparatus according to claim 3, wherein: generate enhanced image features of the encoder layers based on the main fusion features. the first gradient-guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the gradient map of the amplitude-enhanced low-light image and image features of the low-light image: perform layer normalization processing on the image features; perform layer normalization processing on the gradient features; concatenate along a channel dimension to create a combined feature representation that combines image information and gradient information; compress the concatenated combined features in scale using a linear neural network to extract image feature channel importance scores of the image features and gradient feature importance scores of the gradient features.

5. The image processing apparatus according to claim 3, wherein: the low-light image enhancement algorithm model comprises: a decoder module comprising: a plurality of decoder layers corresponding to the plurality of encoder layers, the decoder layers configured to fuse image features output by a previous decoder layer and enhanced image features of a corresponding encoder layer to generate fused image features, the decoder layers further comprising: a second gradient guided calibration unit configured to: extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the amplitude enhanced gradient map of the low-light image and the fused image features; generate processed fused image features based on the image channel importance scores and the fused image features; generate processed gradient features based on the gradient channel importance scores and gradient features; generate primary fused features based on the processed fused image features and the processed gradient features, and a residual connection to the fused image features; generate enhanced fused image features based on the primary fused features; the low-light image enhancement algorithm model generates an enhanced visible light image based on the enhanced image features generated by the decoder module.

6. The image processing apparatus according to claim 5, wherein: the second gradient guided calibration unit is configured to perform the following steps to extract image channel importance scores and gradient channel importance scores using a linear neural network based on gradient features of the amplitude enhanced gradient map of the low-light image and the fused image features: perform layer normalization on the fused image features; perform layer normalization on the gradient features; concatenate along the channel dimension to create a combined feature representation that combines image information and gradient information; use a linear neural network to compress the concatenated combined features in scale to extract image feature channel importance scores for the image features and gradient feature importance scores for the gradient features.

7. The image processing apparatus of any one of claims 1 to 6, wherein: the low-light image enhancement algorithm model is trained based on a dataset comprising low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and the normal-light image samples correspond to each other, and the normal-light image samples and the low-light image samples correspond to each other; the loss function of the gradient reconstruction module is established based on the amplitude enhanced gradient map of the low-light image and the gradient map of the normal-light image generated by the gradient reconstruction module.

8. The image processing apparatus of any one of claims 1 to 6, wherein: the low-light image enhancement algorithm model is trained based on a dataset comprising low-light image samples, normal-light image samples, and infrared image samples; the low-light image samples and the normal-light image samples correspond to each other, and the normal-light image samples and the low-light image samples correspond to each other; the loss function of the low-light image enhancement algorithm model is established based on the enhanced visible light image and the normal-light image.

9. The image processing apparatus of claim 8, wherein: the loss function can be used to calculate one or more of a pixel consistency loss, a perception consistency loss, an amplitude spectrum consistency loss, an edge consistency loss, and a gradient consistency loss.

10. The image processing apparatus according to claim 3 or 5, wherein: the linear neural network comprises a global average pooling, a first linear layer, a ReLU activation function, a second linear layer, and a Sigmoid activation function.

11. An air conditioner, comprising: an indoor unit comprising: an infrared imaging device that can sample an infrared image of an air-conditioned room; a camera device that can sample a visible light image of the air-conditioned room, the visible light image comprising a low-light image; a processor comprising: a low-light image enhancement algorithm model comprising: a gradient reconstruction module configured to obtain a gradient map of an amplitude-enhanced low-light image; the low-light image enhancement algorithm model generates an enhanced visible light image based on gradient features of the gradient map of the amplitude-enhanced low-light image and image features of the low-light image through an encoder-decoder architecture; 12. The air conditioner of claim 11, wherein: a controller configured to perform one or more of face recognition, emotion recognition, gender recognition, age recognition, and physical sign recognition based on the enhanced visible light image and control at least operation of the indoor unit based on the recognition result. the gradient reconstruction module comprises: a frequency domain processing branch configured to convert the gradient map of the low-light image from a spatial domain to a frequency domain to extract a phase spectrum and an amplitude spectrum of the low-light image; and a spatial domain processing branch configured to fuse the gradient map of the low-light image with the infrared image in a channel dimension; the gradient reconstruction module adds a residual of the spatial domain processing branch to the amplitude spectrum output by the frequency domain processing branch to obtain an enhanced amplitude spectrum; and obtains the gradient map of the amplitude-enhanced low-light image through inverse Fourier transform between the enhanced amplitude spectrum and an original phase spectrum.

13. An image processing apparatus, comprising: an acquisition unit configured to acquire images of a target scene; the images comprise infrared images and visible light images, and the infrared images and the visible light images are sampled according to a preset time sequence; 14. The image processing apparatus according to claim 13, wherein: a processing unit configured to recognize an emotion of a target object in the images and / or recognize an emotion change of the target object in the images within a set time period. the processing unit comprises: a fusion recognition model configured to fuse an infrared image feature vector extracted by an infrared feature extraction network and a visible light image feature vector extracted based on a visible light feature extraction network to recognize the emotion of the target object in the images and / or recognize the emotion change of the target object in the images within the set time period.

15. The image processing apparatus according to claim 14, wherein: the fusion recognition model is configured to fuse the infrared image feature vector extracted by the infrared feature extraction network and the visible light image feature vector extracted based on the visible light feature extraction network to recognize the emotion change of the target object in the images within the set time period; the processing unit further comprises: an emotion classification model configured to, after the fusion recognition model recognizes the emotion change of the target object in the images within the set time period, recognize the emotion of the target object in the images based on the images.

16. The image processing apparatus according to claim 14, wherein: The fusion recognition model is configured to fuse an infrared image feature vector extracted by an infrared feature extraction network and a visible light image feature vector extracted based on a visible light feature extraction network, and recognize emotion changes of a target object in an image within a set time period. The processing unit further includes: An emotion segmentation model configured to generate an emotion morphology segmentation result of the target object in the image based on the image after the fusion recognition model recognizes the emotion changes of the target object in the image within the set time period.

17. The image processing apparatus according to claim 14, wherein: The fusion recognition model is configured to fuse an infrared image feature vector extracted by an infrared feature extraction network and a visible light image feature vector extracted based on a visible light feature extraction network, and recognize emotion changes of a target object in an image within a set time period. The processing unit further includes: An emotion classification model configured to recognize an emotion of a target object in an image based on the image after the fusion recognition model recognizes the emotion changes of the target object in the image within the set time period; and An emotion segmentation model configured to generate an emotion morphology segmentation result of the target object in the image based on the image after the fusion recognition model recognizes the emotion changes of the target object in the image within the set time period.

18. The image processing apparatus according to claim 15 or 17, wherein: The emotion classification model is a combination of a convolutional neural network and a Swin Transformer network; the image sequentially passes through the convolutional neural network and the Swin Transformer network to extract encoded features, generate an emotion classification probability based on the encoded features, and recognize an emotion of a target object in the image based on the emotion classification probability.

19. The image processing apparatus according to claim 16 or 17, wherein: The emotion segmentation model is a combination of a convolutional neural network and a Swin Transformer network; the image sequentially passes through the convolutional neural network and the Swin Transformer network to extract encoded features, and the encoded features are input to a network combination composed of a transposed convolutional neural network and a Swin Transformer network for decoding to generate an emotion morphology segmentation result of a target object in the image.

20. The image processing apparatus according to any one of claims 14 to 17, wherein: A fusion recognition model configured to fuse an infrared image feature vector extracted by an infrared feature extraction network, a visible light image feature vector extracted based on a visible light feature extraction network, and a detection value feature vector extracted based on sensor detection data, and recognize an emotion of a target object in an image and / or recognize emotion changes of the target object in the image within a set time period.

21. The image processing apparatus according to any one of claims 14 to 17, wherein: The infrared image and the calculated position code are added to each other, and the result is input to the infrared feature extraction network to extract the infrared image feature vector. The visible light image is added to the calculated position code and input to the visible light feature extraction network to extract the visible light image feature vector.

22. The image processing apparatus according to claim 21, wherein: The fusion recognition model is configured to perform the following method to fuse the infrared image feature vector extracted by the infrared feature extraction network and the visible light image feature vector extracted based on the visible light feature extraction network: The infrared image feature vector and the visible light image feature vector are spliced or weighted summed to obtain a fusion feature; An attention mechanism is applied based on the fusion feature to obtain an attention output; The attention output is processed by a sigmoid function to obtain a probability vector; The emotion of the target object in the image and / or the emotion change of the target object in the image within a set time period is recognized based on the probability vector.

23. An air conditioner, comprising: an indoor unit, comprising: an infrared imaging device that can sample and store infrared images of an air-conditioned room in a preset time sequence; a camera device that can sample and store visible light images of the air-conditioned room in a preset time sequence; an indoor fan configured to supply air to the air-conditioned room; a processing unit configured to recognize the emotion of a target object in the image and / or recognize the emotion change of the target object in the image within a set time period; a control unit configured to adjust the operation of the indoor fan and / or adjust the set temperature of the air-conditioned room based on the recognized emotion of the target object and / or the recognized emotion change of the target object within the set time period.

24. The air conditioner of claim 23, wherein: The processing unit comprises: a fusion recognition model configured to fuse the infrared image feature vector extracted by the infrared feature extraction network and the visible light image feature vector extracted based on the visible light feature extraction network, to recognize the emotion of the target object in the image and / or recognize the emotion change of the target object in the image within a set time period.

Images