Small target detection apparatus, method and system for complex scene, and storage medium

Abstract

Disclosed in the present invention are a small target detection apparatus, method and system for a complex scene, and a storage medium. The apparatus comprises: an adaptive enhancement module, which is used for improving the capability of focusing on and extracting key features of different types of targets in a complex scene, comprising performing adaptive enhancement preprocessing on a low-quality image and enhancing key feature extraction of the different types of targets; a feature extraction module, which is used for adaptively mining, from the enhanced image, a multi-scale feature map representing key features of a small target; a feature fusion module, which is used for fusing, on the basis of the multi-scale feature, a shallow feature map containing target detail information and a deep feature map containing category semantic information, and for improving the sensitivity to features of a complex and irregular small target; and a prediction module, which is used for performing target classification and localization on the basis of the fused feature map acquired by the feature fusion module. By using the technical solution of the present invention, the high-precision recognition and localization of small targets are realized, and adaptability to multiple complex scenes is also realized.

Description

Complex scene small target detection device and method, system, storage medium TECHNICAL FIELD

[0001] The present application belongs to the technical field of target detection, and particularly relates to a complex scene small target detection device and method, system, and storage medium. BACKGROUND

[0002] With the rapid development of computer vision technology, target detection has become a core task in this field. Target detection not only needs to locate all targets in the image, but also needs to identify the category of each target. In recent years, thanks to the progress of deep learning technology, target detection algorithms have made significant breakthroughs in accuracy and speed. However, in practical applications, we often face the challenge of complex small target detection. Small targets usually refer to targets that occupy a small pixel area in the image, such as vehicles in remote sensing images, pedestrians in surveillance videos, or animals in outdoor images, industrial product surface defects, and various complex targets in coal mine underground. Due to the uneven scale of small targets, irregular visual features, and small area occupied in the image, in the case of complex environment, poor lighting conditions, and low image quality, there are often problems such as complex background interference and target occlusion, and small target detection in such cases is more difficult than general target detection.

[0003] The algorithms currently applied to target detection can be mainly divided into two categories: single-stage and two-stage target detection algorithms. The single-stage target detection algorithm usually has the advantages of fast speed, simple deployment and high real-time performance. This algorithm can simultaneously realize the regression of the predicted frame and the classification prediction task, realize end-to-end target detection, and the classic single-stage target detection algorithm mainly includes: YOLO (You Only Look Once) series algorithm and SSD (Single Shot Detector) series algorithm. The two-stage target detection algorithm has two stages of candidate frame generation and classification fine adjustment, and can achieve higher accuracy in target positioning and classification, and is more suitable for processing multi-scale targets. However, the two-stage method often has the disadvantages of high complexity, difficulty in deployment and unsuitability for high real-time applications. The classic two-stage target detection algorithm mainly includes: R-CNN (Region with CNN feature) series algorithm and Faster R-CNN algorithm. In view of the detection and positioning requirements of complex small targets in many industrial scenes at present, the high precision, high speed and convenient deployment of the target detection algorithm in limited computing resources of the device have higher requirements, therefore, the single-stage target detection algorithm has more advantages than the two-stage target detection algorithm. In recent years, with the continuous development of the YOLO network, many researchers have designed target detection models applied to different scenes based on the YOLO network, but for the detection method of complex small targets, in addition to the challenges of large algorithm model parameter quantity, long inference time, and unsatisfactory false detection and missed detection rate, it also faces the challenges of being easily disturbed by the environment and difficult to extract target features due to the uneven scale of small targets, the unobvious visual features and the irregularity. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a small target detection device and method, system and storage medium that can adapt to different complex scenes.

[0005] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0006] A complex scene small target detection device, comprising:

[0007] An adaptive enhancement module is configured to perform adaptive enhancement preprocessing on low-quality images in complex small target images and to perform feature extraction enhancement on key features of different types of targets;

[0008] A feature extraction module is configured to adaptively mine multi-scale feature maps representing key features of small targets from the enhanced images;

[0009] A feature fusion module is configured to fuse shallow feature maps containing target detail information and deep feature maps containing category semantic information according to the multi-scale features obtained from the feature extraction module.

[0010] a prediction module for target classification and positioning according to the fusion feature map obtained from the feature fusion module.

[0011] As preferred, the feature extraction module is constructed based on a designed DSC_Attention module, a two-dimensional convolution module Conv, a dynamic snake convolution module DSConv, an Attention module and an SPP module. Among them, the embedding of the attention mechanism in DSC_Attention and Attention adopts an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is a CPCA attention mechanism module.

[0012] As preferred, the adaptive enhancement module includes a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module of a multi-attention mechanism adaptive embedding network based on a heat map and DDQN.

[0013] As preferred, the feature fusion module contains 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample up-sampling modules; among them, the fusion path from bottom to top in the feature fusion module includes 3 SDFM modules respectively placed in L13, L16 and L19 layers; 3 DSCSP modules are placed in L14, L17 and L20 layers after the 3 SDFM modules; L12, L15 and L18 are Upsample up-sampling modules; the path from top to bottom includes 3 groups of same module structure: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

[0014] The application also provides a complex scene small target detection method, comprising:

[0015] The adaptive enhancement module is used for adaptive enhancement preprocessing of low-quality images in the complex small target image and feature extraction enhancement of key features of different types of targets;

[0016] The feature extraction module is used for adaptive mining of multi-scale feature maps representing key features of small targets from the images after enhancement processing;

[0017] The feature fusion module is used for fusion of shallow feature maps containing target detail information and deep feature maps containing category semantic information according to the multi-scale features obtained from the feature extraction module;

[0018] The prediction module is used for target classification and positioning according to the fusion feature map obtained from the feature fusion module.

[0019] As preferred, the feature extraction module is constructed based on a designed DSC_Attention module, a two-dimensional convolution module Conv, a dynamic snake convolution module DSConv, an Attention module and an SPP module. Among them, the embedding of the attention mechanism in DSC_Attention and Attention adopts an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is a CPCA attention mechanism module.

[0020] As preferred, the adaptive enhancement module includes a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module of a multi-attention mechanism adaptive embedding network based on a heat map and DDQN.

[0021] As preferred, the feature fusion module contains 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample up-sampling modules; among them, the fusion path from bottom to top in the feature fusion module includes 3 SDFM modules respectively placed in L13, L16 and L19 layers; 3 DSCSP modules are placed in L14, L17 and L20 layers after the 3 SDFM modules; L12, L15 and L18 are up-sampling modules; the path from top to bottom includes 3 groups of same module structures: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

[0022] The embodiment of the present application also provides a complex scene small target detection system, comprising a memory and a processor, the memory is stored with a computer program run by the processor, and the computer program executes the complex scene small target detection method when being run by the processor.

[0023] The embodiment of the present application also provides a storage medium, the storage medium is stored with a computer program, and the computer program executes the complex scene small target detection method when being run.

[0024] Based on the research on deep reinforcement learning, dynamic snake convolutional neural network, attention mechanism embedding, multi-scale feature efficient fusion and small target detection layer optimization, the present application designs a complex scene small target detection method based on reinforcement learning adaptive enhancement, realizes high-precision identification and positioning of small targets while having adaptive ability to multiple complex scenes, and has the following technical effects:

[0025] (1) To effectively extract the key features of small and irregularly distributed targets in complex environment images, improve the positioning accuracy and detection sensitivity of small targets, a complex small target feature extraction module based on dynamic snake convolution and attention mechanism (DSAM) is proposed. This module introduces dynamic snake convolution to expand the receptive field and enhance the network's ability to extract complex and irregular target features. By pre-embedding the CPCA attention mechanism module, the ability of the feature extraction network to focus on small target details is improved, and the interference of complex background is suppressed to improve the detection sensitivity of the target.

[0026] (2) To realize adaptive low-quality images collected in different environments during target detection, improve detection accuracy and generalization performance of the detection model, a complex scene target image adaptive enhancement module based on DDQN is designed. Based on four image enhancement processing submodules (image denoising processing, defogging processing, brightness enhancement, and super-resolution), the module combines the DDQN-based reinforcement learning optimization model to adaptively select the optimal image enhancement method combination for low-quality image enhancement processing.

[0027] (3) To realize the feature extraction module's ability to adapt to different types of target features in complex environments, a feature extraction network multi-attention mechanism adaptive embedding module based on heat map and DDQN is proposed. The module uses an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, combined with five alternative attention mechanism modules (channel attention mechanism, spatial attention mechanism, coordinate attention mechanism, CBAM attention mechanism, and EMA attention mechanism module), to adaptively adjust the attention mechanism in the feature extraction module when dealing with different types of targets in other complex environments, achieving better target key feature focusing and extraction performance.

[0028] (4) For different levels of multi-scale features obtained from the feature extraction module, to efficiently fuse shallow features containing detailed information and deep features containing semantic information, a feature fusion module that combines dynamic snake convolution and multi-attention mechanism, MBiFPN, is proposed. Based on the BiFPN structure, the module introduces dynamic snake convolution, embedding semantic and detail information fusion modules with coordinate attention and spatial attention mechanisms, to enhance multi-scale feature fusion performance while improving the model's focusing ability and detection sensitivity for complex small targets, reducing target false positives and missed detections.

[0029] (5) In order to further improve the positioning accuracy of the target detection model for complex small targets and accelerate the convergence speed of the loss function, four new independent decoupling detection heads DSCSHead are designed to perform the positioning regression and identification classification tasks of the target detection model. In the DSCSHead, dynamic snake convolution is introduced to improve the positioning accuracy of complex small targets, and SIoU is used to improve the regression loss function to accelerate the model convergence. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0031] Fig. 1 is a structural schematic diagram of a complex scene small target detection device according to an embodiment of the present application;

[0032] Fig. 2 is an internal structure of the DSC_Attention module;

[0033] Fig. 3 is a structure diagram of the DSAM module;

[0034] Fig. 4 is a LIAENDRL structure;

[0035] Fig. 5 is a motion space diagram of the optimal enhancement strategy learning of low-quality images based on DDQN;

[0036] Fig. 6 is a historical action vector diagram;

[0037] Fig. 7 is a MAMHD structure;

[0038] Fig. 8 is a motion space diagram of the optimal attention mechanism embedding strategy learning based on DDQN;

[0039] Fig. 9 is a historical action vector diagram;

[0040] Fig. 10 is a structure diagram of the SDFM module;

[0041] Fig. 11 is a structure diagram of the DSCSP module;

[0042] Fig. 12 is a structure diagram of the DSCSHead module;

[0043] Fig. 13 is a diagram of related parameters of the SIoU loss function. DETAILED DESCRIPTION

[0044] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of the present application.

[0045] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0046] Embodiment 1

[0047] As shown in FIG. 1, the embodiment of the present application provides a complex scene small target detection device, which comprises:

[0048] An adaptive enhancement module is configured to perform adaptive enhancement preprocessing on low-quality images in complex small target images and to perform feature extraction enhancement on key features of different types of targets.

[0049] A feature extraction module DSAM is configured to adaptively mine a multi-scale feature map representing key features of small targets from the images after enhancement processing.

[0050] A feature fusion module MBiFPN is configured to fuse shallow feature maps containing target detail information and deep feature maps containing category semantic information according to the multi-scale features obtained from the feature extraction module.

[0051] A prediction module is configured to perform target classification and positioning according to the fusion feature maps obtained from the feature fusion module.

[0052] As an implementation manner of the embodiment of the present application, the working process of the adaptive enhancement module is as follows:

[0053] In order to improve the adaptive ability of the target detection model to different types of targets in different complex scenes, the adaptive ability of the detection model is improved from the aspects of image enhancement preprocessing and feature extraction. For image preprocessing, a complex scene target image adaptive enhancement preprocessing module is designed to perform adaptive enhancement preprocessing on low-quality images in different complex scenes. For feature extraction enhancement, based on the optimal attention mechanism adaptive embedding strategy of deep reinforcement learning, a feature extraction adaptive enhancement module based on heat map and DDQN is proposed to adaptively select the attention mechanism more conducive to key feature extraction and embed it into the feature extraction module, so as to improve the key feature extraction ability of the feature extraction module to different types of targets.

[0054] Complex scene target image adaptive enhancement preprocessing module

[0055] In many complex environments, poor lighting conditions and limited image acquisition device performance, the target image obtained will face the problem of poor quality, blurred target, low resolution, and non-obvious visual features, which will seriously affect the accuracy of target detection. To this end, in order to adapt to the low-quality image target detection collected in different environments, improve the detection accuracy and the generalization performance of the detection model, a low-quality image preprocessing module is designed, a low-quality image adaptive enhancement network based on deep reinforcement learning (LIAENDRL), which is an image enhancement strategy that is beneficial to the improvement of detection accuracy, and converts low-quality images into high-quality images, and finally improves the target feature extraction effect of the model on low-quality images.

[0056] The image enhancement preprocessing module LIAENDRL designed in the embodiment of the application is shown in FIG. 4, and is specifically described as follows:

[0057] (1) Image enhancement method. In order to adapt to low-quality image enhancement in different scenes, four image enhancement processing sub-modules are used for image denoising processing, fog removal processing, brightness enhancement and super-resolution. Among them, the image denoising adopts a bilateral filtering algorithm to realize the smoothing of the image while keeping the target edge information; the removal processing adopts a dark channel prior de-fogging algorithm; the brightness enhancement adopts a URetinex-Net algorithm; and the image super-resolution adopts an enhanced super-resolution generative adversarial network (ESRGAN).

[0058] (2) The low-quality image enhancement strategy adaptive optimization based on DDQN. When facing low-quality images, experienced experts and researchers usually manually select and adjust image enhancement algorithms and parameters according to their application scenarios, but this manual image enhancement algorithm and parameter adjustment method is difficult to adapt the target detection model to different use scenarios, for example, images in fog, smoke and other scenes need to be focused on defogging processing, images in low light or uneven light distribution scenes need to be focused on brightness enhancement processing, and for low-resolution images, super-resolution reconstruction images are needed to improve the target feature extraction effect. To this end, the embodiment of the application adopts a deep reinforcement learning method DDQN to learn the image enhancement strategy to replace the manual selection of the image enhancement algorithm. As shown in FIG. 4, the low-quality image is input into the DDQN, and the strategy of obtaining the image enhancement processing is output, the optimal algorithm combination is selected from four image enhancement processing methods, the low-quality image is enhanced, and the high-quality image which can effectively improve the target detection accuracy is obtained.

[0059] The network structure of DDQN consists of three parts. The first part is a feature extraction layer composed of a three-layer CNN network. The second part is two sub-networks composed of fully connected layers. One sub-network is a state value function network V(s; θ, β), wherein s represents the state, θ represents the CNN network parameter, and β is the parameter of the state value function network V. The other sub-network is an advantage function network A(s, a; θ, α) that predicts each action, wherein s represents the state, a represents the action to be performed, θ represents the CNN network parameter, and α is the parameter of the advantage function network A. The third part is to combine the results V(s; θ, β) and A(s, a; θ, α) of the two sub-networks of the second part to obtain the final state-action value Q, and its expression is: Q(s, a; θ, α, β) = V(s; θ, β) + A(s, a; θ, α) (15)

[0060] For the training and learning of the low-quality image optimal enhancement strategy based on DDQN, the action with the maximum value Q is selected as the next execution according to the current strategy of the model through continuous contact with the environment. In this process, the specific action reward value given by the environment is used to evaluate the effect of action execution, and through iterative optimization, the learning process is finally terminated under certain conditions, and the optimal strategy is output.

[0061] For the low-quality image optimal enhancement strategy learning model based on DDQN designed by the embodiment of the application, the specific construction of the action space, state space, reward function and training optimization process is as follows:

[0062] ① Action space. The action space records all the behavior set that the DDQN agent can take in the environment of processing low-quality images. The embodiment of the present application designs four image processing modules, each module has two execution actions: processing the module and no operation (not processing the module), therefore, a total of 16 actions can be executed. The schematic diagram of the action space is shown in Figure 5. The history information of the last four selected actions is recorded, as shown in Figure 6, which is a 4x4 vector, denoted as h, and the value is 0 when initialized. A new vector h0 is constructed for the current selected action each time, which is a 1x4 vector, and the action expressed by the vector corresponds to Figure 5. When updating h, first scroll down 1 step in the row dimension, and then assign h0 to the first row of the current h. When the agent selects the termination action, it means that the current state is the best according to the strategy, and no action needs to be taken, which indicates that the action decision iteration process will end, and the current action vector is output as the final result.

[0063] ② State space. The state represents the information of the current environment, which strongly influences the model's decision on the next action, so the state space should contain enough comprehensive information. In the task of low-quality image adaptive enhancement, the input is the low-quality picture I to be enhanced, the execution action a obtained from the DDQN-based low-quality image optimal enhancement strategy learning model, and the corresponding reward signal r after executing the action.

[0064] ③ Reward function. The reward function is the key to guide the DDQN agent to find the best action selection strategy. In the embodiment of the present application, the agent adjusts the image enhancement processing strategy through a series of action sequences to finally determine the optimal image enhancement module combination, therefore, it is hoped that during the training process, the image enhanced according to the strategy will gradually be more conducive to improving the target detection accuracy, at this time, the image enhanced according to the strategy is input into the pre-trained target detection model, and the target detection accuracy obtained is used as the reward function to measure the effectiveness of the agent's action selection. Specifically, if the target detection accuracy obtained by the image enhanced according to the current strategy is greater than the target detection accuracy obtained by the image enhanced according to the previous strategy, it means that the strategy action at this time is more conducive to the improvement of the target detection accuracy, and should be rewarded; on the contrary, it should be punished. By constructing a suitable objective function, the problem can be converted into an optimization problem of maximizing the reward objective function, so that the agent can make decisions in line with expectations. The reward function for each action used in the embodiment of the present application is: t (s t ,s t+1 )=sign(Acc(O t+1 ,M pre )-Acc(O t ,M pre )) (16)

[0065] where O t+1 represents the enhanced image after the current action is performed, O t represents the enhanced image after the last action is performed, M pre is the target detection model pre-trained using low-quality images without enhancement processing, Acc(O t+1 ,M pre ) represents the target detection accuracy obtained after O t+1 inputs M pre . The sign function takes the value 1 or -1 according to whether the value of the internal expression is greater than 0. In order to make the number of selections as small as possible, a different reward function is needed for the termination action, as follows:

[0066] where s1 represents the initial state, the low-quality image is not subjected to image enhancement processing, I is the low-quality image; when the target detection accuracy obtained by the enhanced image after the current action is performed is greater than the target detection accuracy of the original low-quality image by τ, the reward η is obtained, otherwise the reward is -η.

[0067] (4) Training optimization. The training process of DDQN is as follows: 100 low-quality images with low illumination, noise, blur and low resolution are selected to form a training set, and 100 high-quality images are selected to form training data. In addition, the "exploration and utilization" method is introduced, and the target network and the experience replay pool are combined to improve the training efficiency and model performance. First, an experience replay pool R is established for model training, and the state, action, reward and next state of each time step are recorded as an experience storage. When the experience pool is mixed with past and recent experiences, the experience samples randomly sampled from R are small in relevance, and these experiences can be reused. Then, the training network Q(s, a; θ) is used to guide action decision, and the target network Q - (s′,a′;θ′) is used to estimate the target Q value; the training network and the target network have the same structure, and the target network needs to be updated directly by obtaining the parameters of the current training network to reduce the amount of calculation; the update process of the training network is completed by minimizing the loss function of formula (3-12): L(s, a|θ) = (y-Q(s, a|θ)) 2 (18)

[0068] In the formula, y represents a reward value, when a non-terminal action, y is the superposition of an immediate reward and a discount value of a future return; when a terminal action, y is equal to the immediate reward, and gamma is a discount factor for determining the proportion of the future return in the cumulative return, and gamma ranges from 0 to 1. In order to learn more experience, an epsilon-greedy strategy is used to determine the action, and the optimal action is performed with a probability of 1-epsilon, and other actions are performed with a probability of epsilon, and the formula is as follows:

[0069] Feature extraction adaptive enhancement module

[0070] In order to improve the key feature focusing and extraction ability of the target detection network on different types of targets (scale, shape, etc.), a feature extraction adaptive enhancement module is designed, and a multi-attention mechanism adaptive embedding network (MAMHD) for feature extraction module based on heatmaps and DDQN is used to adaptively select the attention mechanism more conducive to key feature extraction and embed it into the feature extraction network when facing different types of targets, and improve the key feature extraction ability of the feature extraction module on different types of targets.

[0071] The MAMHD structure designed in the embodiment of the application is shown in Figure 7, and the specific description is as follows:

[0072] (1) Multi-attention mechanism module. In order to adapt to the key feature focusing and extraction of different types of targets in complex scenes, the default embedded attention mechanism module in the feature extraction module is a CPCA attention mechanism module, which involves modules including: DSC_Attention module and Attention module; in addition, a plurality of different types of attention mechanism modules are used as candidate modules for embedding the feature extraction module, including channel attention mechanism module, spatial attention mechanism module, coordinate attention mechanism module, CBAM attention mechanism module and EMA attention mechanism module.

[0073] (2) Optimization of Attention Mechanism Embedding Strategy Based on DDQN. Since the targets to be detected in complex scenes often exhibit significant differences, such as scale imbalance and diverse shapes, the information that needs to be focused on differs for different types of targets during feature extraction. When facing different types of target images, selecting a suitable attention mechanism module can be more beneficial for focusing and extracting key features, thereby improving detection accuracy. In this embodiment, the Deep Reinforcement Learning method DDQN is used to learn the optimal attention mechanism embedding strategy, enabling the adaptive selection of an appropriate attention mechanism module to be embedded into the feature extraction module when facing different types of target images. As shown in Figure 7, the image to be detected is input into DDQN, and the output obtains the attention mechanism embedding strategy. The module most suitable for extracting key features from the image to be detected is selected from five attention mechanism modules and embedded into the feature extraction module, enhancing the ability to focus and extract key features.

[0074] This uses the same DDQN network structure as the LIAENDRL network, and the training and optimization process follows the same principle. The main differences lie in the action space, state space, and reward function.

[0075] ① Action Space. The action space records the set of all actions that the DDQN agent can take in the environment. This embodiment of the invention designs 5 attention mechanism embedding modules, each with 2 execution actions: performing the module's processing and no operation (not embedding the module). Since the final selected attention mechanism module is either one or none, there are a total of 6 actions. A schematic diagram of the action space is shown in Figure 8. Figure 9 shows the historical information of the 4 most recently selected actions, forming a 4×6 vector, using one-hot encoding, denoted as g, with initial values ​​of 0. Each time, a new vector h0 is constructed for the currently selected action. This new vector has a dimension of 1×6, and the action represented by the vector corresponds to that in Figure 8. Whenever g is updated, it first scrolls down one row dimension, and then assigns g0 to the first row of the current g. When the agent chooses to terminate the action, it means that the current state is optimal according to the policy, and no action needs to be taken. This indicates that the action decision iteration process will end, and the current action vector is output as the final result.

[0076] ② State Space. The state represents information about the current environment and strongly influences the model's decision on the next action; therefore, the state space should contain sufficient comprehensive information. In the task of adaptive learning of the optimal attention mechanism embedding policy, the input is the image to be detected I, the action b obtained from the DDQN-based optimal attention mechanism embedding policy learning model, and the corresponding reward signal r after the action is executed.

[0077] ③ Reward Function. The reward function is key to guiding the agent to find the optimal action selection strategy. In this embodiment, the agent adjusts the attention mechanism embedding strategy through a series of action sequences, ultimately determining the optimal attention mechanism to be embedded in the feature extraction module. This embodiment obtains a heatmap of the image to be detected from the feature extraction module and uses it to determine the candidate localization boxes of the target to be detected. It is hoped that during the training process, the feature extraction module that embeds the attention mechanism according to the strategy can effectively adapt to the feature extraction of different types of targets, obtain more accurate key feature information, and thus improve the detection accuracy. At this time, the image to be detected is input into the feature extraction module that embeds the corresponding attention mechanism according to the strategy to obtain the heatmap of the image. The candidate localization boxes are determined based on the range where the pixel value in the heatmap is greater than σ times the maximum pixel value (σ∈[0.1,1]). The IoU between the candidate localization box and the calibration box of the image to be detected is calculated as the reward function to measure the effectiveness of the agent's action selection. Specifically, if the feature extraction module that embeds the corresponding attention mechanism according to the current strategy can increase the IoU between the candidate localization box and the calibration box of the image to be detected, it indicates that the current strategy action is more conducive to the key feature extraction of the target to be detected and should be rewarded; conversely, it should be penalized. By constructing an appropriate objective function, the problem can be transformed into an optimization problem of maximizing the reward, enabling the agent to make decisions as expected. The reward function for each action used in this embodiment of the invention is as follows:

[0078] In the formula, Box t+1 Box represents the candidate localization box of the target object determined from the heatmap of the image to be detected obtained by the feature extraction module after the current action is executed. t Box represents the candidate localization box of the target object determined from the heatmap of the image to be detected obtained by the feature extraction module after the current action is executed. pre The bounding box for the image to be detected, IoU (Box) t+1 Box pre ) represents Box t+1 With Box pre The IoU between them. The sign function takes a value of 1 or -1 based on whether the value of the internal expression is greater than 0. To minimize the number of selections, different reward functions are needed for the termination action, as follows:

[0079] In the formula, s1 represents the initial state, where the initial attention mechanism in the feature extraction module has not been replaced, and Box1 represents the candidate localization box of the target object determined by the heatmap of the image to be detected obtained from the feature extraction module with the embedded default attention mechanism; after this action is executed, IoU(Box1) = 1 / 2 * ... t+1 Box pre-IoU(Box1,Box pre When the value is greater than T, the reward is η; otherwise, the reward is -η.

[0080] ④ Training Optimization. The training process of DDQN here is the same as that of the LIAENDRL network. It still introduces the "exploration and exploitation" approach and combines the target network and the experience replay pool to improve training efficiency and model performance. The training dataset consists of 100 images each from two scenarios (images of small, easily occluded targets in underground coal mines and images of steel strip defects) containing different types of targets.

[0081] As one embodiment of the present invention, the workflow of the feature extraction module DSAM is as follows:

[0082] In the feature extraction module DSAM proposed in this embodiment of the invention, dynamic serpentine convolution is introduced to enhance the ability to accurately capture the features of complex small targets. At the same time, a channel-priority convolutional attention mechanism is embedded to achieve adaptive attention to channel and spatial information in the process of extracting features of complex small targets. This promotes rapid focusing on important information channels and important spatial regions, better suppresses the interference of background noise, and enhances the ability of the extracted features to express the position and shape of the target. In this way, the target detection network can improve the accuracy of locating different complex small targets.

[0083] (1) Introduce dynamic snake convolution to enhance the feature extraction module’s ability to accurately capture features of complex small targets.

[0084] Complex small targets often exhibit a variety of locations and shapes, resulting in complex and irregular features in their images. Since most object detection models based on Convolutional Neural Networks (CNNs) use fixed convolution kernels, their receptive fields are limited, restricting their ability to capture a large amount of local information. Therefore, this traditional fixed convolutional structure limits the extraction of complex and irregular features, increasing the difficulty of detecting complex small targets and reducing the model's generalization ability. To enhance the accuracy of CNN-based object detection models in detecting complex small targets, adaptively expanding the receptive field is crucial. This helps the detection model learn long-distance and irregular spatial relationships, thereby improving its feature extraction performance. To address this, this invention introduces Dynamic Snake Convolution (DSConv), which learns deformations from the input feature map and adaptively focuses on tortuous and elongated local structures of the target. By expanding the local receptive field, it more accurately captures features of irregular shapes and structures, improving the detection model's ability to adapt to different target geometries and sizes, thus promoting the model's flexible recognition of irregular features in images of complex small targets.

[0085] This invention introduces the DSC_Attention module into the feature extraction module to enhance the adaptive feature extraction capability for complex small targets. Figure 2 shows the structure of the DSC_Attention module. The DSC_Attention module is an improvement on the C2f module in the YOLO series algorithms. A new DCSA module is designed to replace the bottleneck module in the original C2f module. The DCSA module includes two DSConv modules and one attention mechanism module.

[0086] (2) Introduce an attention mechanism to improve the detection model’s ability to accurately locate small targets.

[0087] Complex small targets, due to their small scale, irregularity, and uneven distribution, are easily affected by background images and external noise, leading to a decrease in the accuracy of many existing detection models in locating complex small targets. In recent years, many researchers have proposed various forms of attention mechanisms, such as channel attention and spatial attention, to improve the ability of detection models to focus on key features of targets. However, considering channel attention or spatial attention alone only improves the detection effect for conventional targets to a certain extent; however, the detection effect is still unsatisfactory when facing complex small targets that are easily affected by the environment. To address this, this embodiment of the invention introduces a channel-prior convolutional attention (CPCA) mechanism by default in the DSC_Attention module and the Attention module. This enables the dynamic distribution of attention weights in the channel and spatial dimensions while combining channel attention and spatial attention mechanisms, promoting rapid focusing on important information channels and important spatial regions, and improving the accuracy of the target detection network in locating small targets. Furthermore, to further improve the adaptability of the feature extraction module designed in this embodiment of the invention when dealing with feature extraction of different types of targets, an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning is also designed. This strategy enables adaptive adjustment of the attention mechanism module in the feature extraction module when dealing with different types of targets in other complex environments, thereby achieving more ideal focus and extraction of key target features. The DSC_Attention module and the Attention module in the feature extraction module are initially embedded with the CPCA attention mechanism module by default.

[0088] (3) Construct a feature extraction module DSAM based on dynamic snake convolution and attention mechanism.

[0089] To effectively enhance the ability of feature extraction networks to accurately capture the location and shape features of complex small targets, this invention designs and constructs a feature extraction module DSAM based on traditional Conv, DSConv, DSC_Attention, Attention, and SPP modules. The structure of this module is shown in Figure 3. Layers L1-L11 in the figure represent the sub-modules of the DSAM module, comprising two Conv modules, three DSConv modules, four DSC_Attention modules, one Attention module, and one SPP module. The DSC_Attention module and the Attention module initially embed the CPCA attention mechanism module by default. Layers L3, L5, L7, and L11 in this module can output feature maps of different scales, which are then input to the feature fusion network MBiFPN for multi-scale feature fusion.

[0090] As one embodiment of the present invention, the workflow of the feature fusion module MBiFPN is as follows:

[0091] To better and more efficiently utilize the feature information contained in multi-scale feature maps that is beneficial for the classification, recognition, and localization of complex small targets, a new feature fusion module, MBiFPN, was designed and constructed based on the BiFPN structure. This module incorporates a semantic and detail information fusion module (SDFM), a DSCSP module, an Upsample module, and a dynamic snake convolution module. This allows for the efficient fusion of shallow feature maps containing target detail information and deep feature maps containing category semantic information, while improving the sensitivity to the localization of complex small targets.

[0092] (1) A semantic and detail information fusion module with embedded coordinate attention and spatial attention mechanisms is introduced to enhance the performance of multi-scale feature fusion. To better and more efficiently utilize the feature information contained in multi-scale feature maps that is beneficial for target detection and recognition, an SDFM module is designed and introduced into the BiFPN module to fuse feature maps of different scales. The structure of the SDFM module is shown in Figure 10. The figure shows the internal structure of the SDFM module taking the fusion of 3 layers of feature maps at different scales as an example. Assume the M-layer feature map is... in, 1≤m≤M represents the feature map of the m-th layer. Based on the SDFM module structure diagram shown in Figure 10, First, the feature map is processed by spatial attention and coordinate attention modules to obtain feature maps containing local spatial information and global channel information. Then use dynamic serpentine convolution to process. Reduce its number of channels to obtain the feature map Then, the size of each feature map is resized to match the feature maps. The resolution is expressed as follows:

[0093] In the formula, F I ,F D and F U They represent respectively to Perform identity mapping, adaptive average pooling, and bilinear interpolation to match its resolution to H. m ×W m Based on this, we further utilize 3×3 convolution pairs Perform smooth convolution operation to obtain The expression is as follows:

[0094] Where, ρ mn The parameters represent the smoothing operation. This represents the nth smooth feature map at layer m. Finally, to further enhance the integration of more target semantic information and target boundary details into the feature map at layer m, the Hadamard product is used to process the feature map at layer m, as shown in the following expression:

[0095] In the formula, H(·) represents the Hadamard product operation.

[0096] (2) Dynamic serpentine convolution is introduced to enhance the target feature extraction capability and improve the sensitivity to complex and irregular small target features. In this embodiment of the invention, the DSCSP module is designed and constructed based on the traditional Conv module, DSConv module, and Concat module. This module enhances the target feature extraction capability by introducing dynamic serpentine convolution, while also reducing computational load and network complexity. The structure of the DSCSP module is shown in Figure 11.

[0097] (3) Construct the feature extraction module MBiFPN.

[0098] To achieve efficient fusion of shallow feature maps containing target detail information and deep feature maps containing category semantic information, and to improve sensitivity to complex small target features, this invention designs and constructs a new feature fusion module, MBiFPN, based on the traditional DSConv, DSCSP, SDFM, and Upsample modules, on the BiFPN structure. The structure of this module is shown in Figure 1. The bottom-up fusion path in the MBiFPN network includes: three SDFM modules placed in layers L13, L16, and L19, respectively; three DSCSP modules placed after the three SDFM modules (layers L14, L17, and L20). L12, L15, and L18 are Upsample modules. The top-down path includes three sets of identical module structures: layers L21-L23 are constructed from one DSConv module, one SDFM module, and one DSCSP module; layers L24-L26 and L27-L29 are also constructed in the same way. After constructing the top-down and bottom-up fusion paths in the feature fusion module through the above modules, based on the design concept of BiFPN structure, this embodiment of the invention constructs connections from the feature maps output by L5 and L7 layers to L22 and L25 layers, respectively.

[0099] The multi-scale feature fusion module MBiFPN constructed in this embodiment of the invention can achieve efficient fusion of multi-scale features while further enhancing the sensitivity to feature extraction of complex small targets, thereby improving the accuracy of small target identification and localization.

[0100] As one embodiment of the present invention, based on the four-layer output feature maps (L20, L23, L26, L29) of the feature fusion module MBiFPN, four DSCSHead target prediction branches are designed and constructed. Each branch, based on the independent decoupled detection head in the YOLOv8 series algorithms, introduces dynamic serpentine convolution and improves the bounding box regression task loss function to improve the accuracy of complex small target recognition while accelerating convergence. The structure of the designed DSCSHead module is shown in Figure 12. For the original independent decoupled detection head, when facing small target detection tasks interfered with by complex environmental noise, the slight intra-frame positional deviation of the predicted small pixel target can adversely affect the calculation of the bounding box loss function, easily leading to a decrease in recognition accuracy; and when facing small targets with insignificant aspect ratio or scale changes, the convergence rate of the loss function is not high, thus affecting the performance of the entire regression task. To overcome the above problems, this embodiment of the present invention introduces dynamic serpentine convolution and improves the bounding box regression task loss function, replacing the CIoU loss function with the SIoU loss function and combining it with DFL to perform the bounding box regression task; the BCE loss function is used for the classification branch to handle the target classification task.

[0101] SIoU considers four cost factors in constructing the objective loss function: Angle cost, Distance cost, Shape cost, and IoU cost. Figure 13 shows the parameters related to the SIoU loss function.

[0102] (1) The expression for Angle cost is as follows: E angle =1-2×sin 2 (arcsin(C h / σ)-(π / 4)) (30)

[0103] In the formula,

[0104] (2) The expression for Distance cost is as follows:

[0105] (3) The definition of Shape cost is:

[0106] (4) The definition of IoU Cost is:

[0107] In the formula, B and B GT These represent the true calibration box and the predicted box, respectively.

[0108] (5) SIOU Loss is defined as follows:

[0109] Based on the definition of the SIoU loss function, the distance cost calculation considers the angular loss related to the ground truth bounding box and the predicted bounding box. Introducing angular cost into the loss function facilitates the calculation of the distance cost between the bounding box and the predicted bounding box. In the initial stage of detection model training, the predicted bounding box and the bounding box usually do not intersect. By introducing angular cost, the calculation of the distance between the two types of boxes can be accelerated, thereby achieving faster convergence of the entire regression task. In summary, with the introduction of angular cost, the SIoU loss function achieves a more comprehensive loss function calculation, while further improving the stability of the loss function convergence, thus improving regression accuracy and reducing prediction error.

[0110] To overcome the problems of low detection accuracy and poor model robustness in complex environments, poor lighting conditions, and low image quality caused by the uneven scale and irregular visualization features of small targets, a new detection device for small targets in complex scenes is proposed. This device can achieve high-precision recognition and localization of small targets while having the ability to adapt to multiple complex scenes. The detection device includes four aspects: (1) a complex small target feature extraction module based on dynamic snake convolution and attention mechanism, which effectively enhances the network's ability to extract complex and irregular target features, improves the ability to focus on the detailed features of small targets, and suppresses the interference of complex backgrounds, thereby improving the detection sensitivity of small targets. (2) An image enhancement preprocessing and feature extraction adaptive enhancement module based on reinforcement learning is first used to adaptively select the optimal combination of image enhancement methods to enhance the input low-quality image. Then, the feature extraction module based on heatmap and DDQN is used to adaptively select the attention mechanism that is more conducive to the extraction of key features for different types of targets and embed it into the feature extraction module to improve the feature extraction module's ability to focus on and extract key features of different types of small targets. (3) A feature fusion module that integrates dynamic snake convolution and multi-attention mechanism is used to achieve efficient fusion of shallow features containing detailed information and deep features containing semantic information, thereby improving the sensitivity of the localization of complex small targets. (4) The introduction of dynamic snake convolution and SIoU to improve the detection head further improves the localization accuracy of the target detection model for complex small targets and accelerates the convergence speed of the loss function.

[0111] Example 2:

[0112] This invention also provides a method for detecting small targets in complex scenes, including:

[0113] The adaptive enhancement module performs adaptive enhancement preprocessing on low-quality images containing complex small targets and extracts and enhances key features of different types of targets.

[0114] The feature extraction module adaptively mines multi-scale feature maps representing key features of small targets from the enhanced image;

[0115] The feature fusion module fuses shallow feature maps containing target detail information and deep feature maps containing category semantic information based on the multi-scale features obtained from the feature extraction module.

[0116] The prediction module classifies and locates targets based on the fused feature map obtained from the feature fusion module.

[0117] In one embodiment of the present invention, the feature extraction module is constructed based on a fused dynamic serpentine convolution DSC_Attention module, a two-dimensional convolution module Conv, a dynamic serpentine convolution module DSCConv, an Attention module, and an SPP module. Specifically, the attention mechanism embedding in the DSC_Attention module and the Attention module employs an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, with the initial default embedding attention mechanism module being the CPCA attention mechanism module.

[0118] As one embodiment of the present invention, the adaptive enhancement module includes: an image preprocessing module based on a low-quality image adaptive enhancement network using deep reinforcement learning, and a feature extraction adaptive enhancement module based on a heatmap and DDQN multi-attention mechanism adaptive embedding network.

[0119] In one embodiment of the present invention, the feature fusion module includes: 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules, and 3 Upsample modules. The bottom-up fusion path in the feature fusion module includes: 3 SDFM modules placed in layers L13, L16, and L19 respectively; 3 DSCSP modules placed in layers L14, L17, and L20 after the 3 SDFM modules; and Upsample modules in layers L12, L15, and L18. The top-down path includes three identical module structures: layers L21-L23 are constructed using one DSConv module, one SDFM module, and one DSCSP module; layers L24-L26 and L27-L29 are constructed in the same manner. After constructing the top-down and bottom-up fusion paths in the feature fusion module using the above modules, based on the BiFPN structure design, this embodiment of the present invention constructs connections from the feature maps output by layers L5 and L7 to layers L22 and L25 respectively.

[0120] Example 3:

[0121] This invention also provides a small target detection system for complex scenes, including: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a small target detection method for complex scenes when run by the processor.

[0122] Example 4:

[0123] This invention also provides a storage medium storing a computer program that executes a method for detecting small targets in complex scenes during runtime.

[0124] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A complex scene small target detection device, characterized in that, Comprise: An adaptive enhancement module for adaptive enhancement preprocessing of low-quality images in complex small target images and feature extraction enhancement of key features of different types of targets; A feature extraction module for adaptively mining multi-scale feature maps representing key features of small targets from the enhanced images; A feature fusion module for fusing shallow feature maps containing target detail information and deep feature maps containing category semantic information according to the multi-scale features obtained from the feature extraction module; A prediction module for target classification and positioning according to the fusion feature maps obtained from the feature fusion module.

2. The complex scene small target detection apparatus of claim 1, wherein, The feature extraction module is constructed based on the designed DSC_Attention module, two-dimensional convolution module Conv, dynamic snake convolution module DSConv, Attention module and SPP module. Among them, the embedding of the attention mechanism in the DSC_Attention module and the Attention module adopts an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is a CPCA attention mechanism module.

3. The complex scene small target detection apparatus of claim 2, wherein, The adaptive enhancement module includes a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module of a multi-attention mechanism adaptive embedding network based on a heat map and DDQN.

4. The complex scene small target detection apparatus of claim 3, wherein, The feature fusion module includes 3 DSConv modules, 6 DSCSP modules, 6 SDFM modules and 3 Upsample up-sampling modules; Among them, the fusion path from bottom to top in the feature fusion module includes: 3 SDFM modules are placed in L13, L16 and L19 layers respectively; 3 DSCSP modules are placed in L14, L17 and L20 layers after the 3 SDFM modules; L12, L15 and L18 are up-sampling modules; The path from top to bottom includes 3 groups of same module structure: L21-L23 layers are constructed by 1 DSConv module, 1 SDFM module and 1 DSCSP module, L24-L26 layers and L27-L29 layers are also constructed in the same way.

5. A method for detecting small targets in complex scenes, characterized in that Comprise: Through the adaptive enhancement module, the low-quality images in the complex small target images are adaptively enhanced and preprocessed, and the key features of different types of targets are extracted and enhanced; Through the feature extraction module, multi-scale feature maps representing key features of small targets are adaptively mined from the enhanced images; Through the feature fusion module, shallow feature maps containing target detail information and deep feature maps containing category semantic information are fused according to the multi-scale features obtained from the feature extraction module; Through the prediction module, target classification and positioning are performed according to the fusion feature maps obtained from the feature fusion module.

6. The complex scene small target detection method of claim 5, wherein, The feature extraction module is constructed based on a designed DSC_Attention module, a two-dimensional convolution module Conv, a dynamic snake convolution module DSConv, an Attention module and an SPP module. The embedding of the attention mechanism module in the DSC_Attention and Attention is implemented by using an optimal attention mechanism adaptive embedding strategy based on deep reinforcement learning, and the initial default embedded attention mechanism module is a CPCA attention mechanism module.

7. The complex scene small target detection method of claim 6, wherein, The adaptive enhancement module includes a low-quality image adaptive enhancement network based on deep reinforcement learning and a feature extraction adaptive enhancement module of a multi-attention mechanism adaptive embedding network based on a heat map and DDQN.

8. The complex scene small target detection method of claim 7, wherein, The feature fusion module includes three DSConv modules, six DSCSP modules, six SDFM modules and three Upsample up-sampling modules. In the feature fusion module, the fusion path from bottom to top includes three SDFM modules placed at L13, L16 and L19 layers, respectively, and three DSCSP modules placed at L14, L17 and L20 layers after the three SDFM modules, respectively. L12, L15 and L18 are Upsample up-sampling modules. The path from top to bottom includes three groups of the same module structure: L21-L23 layers are constructed by one DSConv module, one SDFM module and one DSCSP module, and L24-L26 layers and L27-L29 layers are also constructed in the same way.

9. A complex scene small target detection system, characterized in that, The computer program is stored on the storage medium and executed to perform the complex scene small target detection method according to any one of claims 5-8. The computer program is stored on the storage medium and executed to perform the complex scene small target detection method according to any one of claims 5-8.

10. A storage medium, characterized by ​

Images