Bit-plane-based auxiliary single-object tracking adversarial defense reconstruction method

Abstract

A bit-plane-based auxiliary single-object tracking adversarial defense reconstruction method, comprising the steps of: acquiring a video frame sequence composed of clean samples; generating an adversarial example sequence; generating a quantized sample sequence by means of a bit-plane decomposition method; constructing a defense network that outputs a defended sample sequence; acquiring a tracker; jointly training the defense network together with the tracker to update a network parameter of the defense network and obtain a defense reconstruction model; and generating, on the basis of the defense reconstruction model, a defended sample sequence of a video frame sequence to be recognized for tracking by the tracker. The method enhances tracker defense capabilities against unknown perturbances without reducing clean sample accuracy, thereby improving target tracking accuracy, and resolving the problems of high training costs and high computing costs of existing defense methods. Moreover, defense effectiveness is transferable to other CNN- or CNN-Transformer-based trackers in a plug-and-play manner, thus exhibiting good transferability.

Description

An auxiliary single-target tracking adversarial defense reconstruction method based on bit planes TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an auxiliary single-target tracking adversarial defense reconstruction method based on bit planes. BACKGROUND

[0002] 0002.Video single-target tracking is one of the important tasks in computer vision, which refers to using a single-target tracker to extract target visual features based on a specified target region in the initial frame, and to predict the location of the target in the subsequent video frame sequence. Existing single-target trackers are mainly divided into three categories: CNN-based trackers, trackers combining CNN and Transformer, and Transformer trackers.

[0003] 0003.Adversarial attacks mainly add subtle perturbations that are difficult for people to detect in clean samples, so that the normal training of deep learning models outputs high-confidence incorrect predictions. The adversarial attack technology in the target tracking task is to generate imperceptible adversarial perturbations by carefully designed target functions to guide network models or iterative methods to add them to video frames, so that advanced trackers lose the tracking target in the normal tracking process. For example, CSA attack, IoU attack, DFA attack, etc. Adversarial defense aims to eliminate the threat of adversarial samples. At present, there are the following methods:

[0004] Sravanti et al. adopted a bit plane-based quantization technique to achieve adversarial defense in the field of image classification. By quantization technology, high-bit plane images are obtained, which are sent into the neural network together with normal clean images for training. By designing a loss function to strengthen the consistency of the neural network's decision on high-bit plane images and normal clean images (i.e., the regularization factor Bit Plane Feature Consistency, BPFC), the neural network is more likely to use high-bit plane information to form a rough prediction and only use low-bit plane information to refine the prediction. This method does not require adversarial samples for training, and the computational cost is smaller than adversarial training, which can improve the robustness of the network. However, this method loses detail information, and its defense effect and adversarial training methods still have a gap. And currently, this bit plane-based defense method is mainly applied to image classification tasks, and when directly migrated to the target tracking task, the defense effect is not good. The reasons are as follows: first, the network structure of target tracking is different from that of the classifier. Second, the input of the target tracking task is a continuous image sequence, which has time sequence information that does not exist in image classification tasks.

[0005] 0004. Achyut et al. constructed representative support datasets for ten categories on the MNIST dataset. When defending against adversarial attacks, they preserved the high plane of the adversarial example and replaced the low plane of the adversarial example with the low plane from the support dataset of the same category to achieve the defense objective. However, this method has poor transferability and practicality, and its effectiveness on different datasets is questionable.

[0006] 0005. Liu et al. used different bit-plane combinations to input bit-plane classifiers for convolution, and then integrated them with the target DNN model for prediction to defend against adversarial attacks. However, this method has a high computational and training cost. Summary of the Invention

[0007] 0006. The purpose of this invention is to provide a bit-plane-based auxiliary single-target tracking adversarial defense reconstruction method that solves the above problems, reduces the accuracy of clean samples, improves the ability to defend against adversarial attacks, and has a low training cost.

[0008] 0007. To achieve the above objectives, the technical solution adopted by the present invention is as follows: an auxiliary single-target tracking adversarial defense reconstruction method based on the bit plane, comprising the following steps;

[0009] S1, Obtain a video frame sequence consisting of clean samples. Where the clean sample of the i-th frame is I i Its label , , I i The classification labels and regression labels, where n is the total number of clean samples;

[0010] S2, in I i Add perturbation Get I i Corresponding adversarial examples , constitute adversarial sample sequences ;

[0011] S3, using a bit-plane layering method to... Quantization is performed to obtain I i Corresponding quantized samples , constitute a quantized sample sequence ;

[0012] S4, construct a defense network M rec , used for input Output defense sample sequence ;

[0013] Therefore, the defense network M recBased on the Unet network, when i = 1, The defense sample output by the Unet network , when i ≥ 2, Input the Unet network and fuse the previous frame quantization sample The defense sample output after ;

[0014] S5, obtain a tracker M based on a twin network F , the feature extraction network of the M F has J layers, and each layer outputs a feature map;

[0015] The M F is used for input and , and for , the M F outputs the feature map of the corresponding jth layer feature extraction network , the classification label and the regression label , for , the M F outputs the feature map of the corresponding jth layer feature extraction network , the predicted classification label and the predicted regression label , 1 ≤ j ≤ J;

[0016] S6, the M rec and the M F compose a total network, and a total loss function L is constructed Tri , including feature loss L fea , defense sample classification loss L cls and defense sample regression loss L reg ;

[0017] S7, preset the iteration round, train the total network with and , and freeze the network parameters of the M F during training to minimize L Tri update the network parameters of the M rec until the iteration round is reached, and the trained M rec is used as a defense reconstruction model ;

[0018] S8, obtain a video frame sequence to be recognized, generate a corresponding quantization sample sequence according to steps S2 and S3, and then generate a defense sample sequence through the defense reconstruction model , which is used for tracker tracking.

[0019] 0008. As preferred: S3 is quantized in

[0020] into first bit plane b1 to eighth bit plane b8; preset quantization factor k, according to formula , wherein, 1≤k<8, j is the index of jth bit plane.

[0021] 0009. As preferred: k=5.

[0022] 0010. As preferred: step S4 includes steps S41-S42;

[0023] S41, obtaining a Unet network including a shrink path and an expansion path, generating quantized sample , the shrink path generates first down-sampling feature to third down-sampling feature , the expansion path generates third up-sampling feature to first up-sampling feature , and and , and , and , jump connection;

[0024] S42, for , when i=1, directly output defense sample through the Unet network;

[0025] when i=2-n, improve the jump connection mode of the Unet network, let and , and , and jump connection, generate defense sample , wherein, , , are respectively the first down-sampling feature, the second down-sampling feature and the third down-sampling feature of the i-1th frame clean sample I i-1 corresponding to the quantized sample .

[0026] ​​​​​​​0011. As a preferred option: In S6, the feature loss L fea Total loss function L Tri It is obtained from the following formula;

[0027] ,

[0028] ,

[0029] In the formula, α, β, and γ are respectively L fea L cls and L reg The hyperparameter balance factor.

[0030] 0012. As preferred: α=0.4, β=0.3, γ=0.3.

[0031] 0013. Preferably: the tracker M F This includes SiamRPN, SiamRPN++, TransT, and STARK.

[0032] 0014. As a preferred option: In S6, the defensive sample classification loss L cls The cross-entropy loss function is used, and the defensive sample regression loss L reg Smooth L1 loss is used.

[0033] 0015. As a preferred option: the training method for one iteration in S7 includes;

[0034] S71, Enter M rec generate ;

[0035] S72, , Enter M F Calculate the total loss function L Tri ;

[0036] S73, using ADAM optimizer via L Tri Update M rec Network parameters.

[0037] 0016.Bit Plane Slicing: For an image, the pixel value of each point can be converted into its binary representation. Since the minimum and maximum values of image pixels are between 0 and 255, a total of 8 bits are needed to represent the binary representation of the pixel value, so the image can be divided into 8 bit planes, and the bit plane is a matrix with element values of 0 or 1, where the first bit plane b1 contains the lowest order bit of all pixels in the image, and the eighth bit plane b8 contains the highest order bit of all pixels in the image. In the adversarial attack of target tracking, the imperceptible perturbation added is usually located in the low bit plane, so the invention removes the first k low bit planes and retains the high bit planes to form the quantized sample From the formula, the higher the bit plane, the greater the weight, the more retained in the quantized sample, and through the formula, the main visual information of the high bit plane can be retained and most of the perturbations can be removed.

[0038] 0017.The tracker described in the invention includes SiamRPN, SiamRPN++, and TransT, which are commonly used trackers in visual target tracking. For example, SiamRPN is a tracker that adds RPN (Region Proposal Network) to the single target tracking SiamFC. SiamRPN++ is an improved tracker based on SiamRPN. TransT (Transformer Tracking) is a feature fusion model based on Transformer, which effectively aggregates the global information of the target and the search area by establishing nonlinear semantic fusion and mining long-distance feature correlation, significantly improving the accuracy of the algorithm. STARK mainly uses the encoder in Transformer to model the global spatiotemporal feature correlation between the target object and the search area, and uses the decoder in Transformer to learn the query embedding to predict the spatial position of the target object.

[0039] 0018.Compared with the prior art, the invention has the following advantages:

[0040] (1) The invention designs a defense reconstruction network for the unique characteristics of video, fully utilizes the double information in the space-time dimension for feature fusion and interaction, enhances the defense ability of the tracker to unknown perturbations while not reducing the accuracy of clean samples, and improves the accuracy of target tracking.

[0041] 0019. The processing of spatial dimension information is based on a bit-plane hierarchical method. This invention employs a bit-plane hierarchical method to remove most of the perturbations. This is because adversarial attacks on target tracking cause trackers to fail by adding imperceptible perturbations, which are typically located in the low bit plane. Therefore, this invention combines a bit-plane hierarchical method to remove most of the perturbations by deleting visual information from the low bit plane, retaining only the main visual information and a small amount of perturbations from the high bit plane. Furthermore, this method enhances the robustness of the tracker by analyzing the distribution of perturbations through bit-plane visual analysis, without needing to train on adversarial examples, thus solving the problems of high training costs and computational expenses in existing defense methods.

[0042] 0020. Time-dimensional information processing is based on the defense network designed in this invention. Since a small amount of perturbation remains in the high-bit plane after using the bit-plane layering method, deleting the low-bit plane will result in the loss of some detailed information, reducing the accuracy of clean samples and limiting the ability to defend against adversarial attacks. Therefore, this invention designs a defense network M. rec This network is used to compensate for the loss of low-bit planes and recover detailed information. Furthermore, it enhances the fusion of feature information, which helps improve defense performance.

[0043] 0021. (2) The defense network designed based on this invention can maintain good generalization against various adversarial attack methods, including black-box and white-box attacks. Compared with adversarial training, this invention maintains better accuracy on clean samples.

[0044] 0022. (3) The defense network trained by this invention can be transferred to other CNN-based or CNN and Transformer-based trackers in a plug-and-play manner without adjusting the tracker parameters, thus exhibiting good transferability. When the defense network is used in combination with other trackers, it does not introduce excessive computational overhead, and the speed is considerable. Attached Figure Description

[0045] 0023. Figure 1 is a structural diagram of the present invention;

[0046] Figure 2 shows the Unet network structure.

[0047] Figure 3 is a flowchart of the defense network workflow of the present invention. Detailed Implementation

[0048] 0024. The present invention will be further described below with reference to the embodiments.

[0049] 0025. Example 1: Referring to Figures 1 to 3, an auxiliary single-target tracking adversarial defense reconstruction method based on the bit plane includes the following steps;

[0050] S1, Obtain a video frame sequence consisting of clean samples. Where the clean sample of the i-th frame is I i Its label , , I i The classification labels and regression labels, where n is the total number of clean samples;

[0051] S2, in I i Add perturbation Get I i Corresponding adversarial examples , constitute adversarial sample sequences ;

[0052] S3, using a bit-plane layering method to... Quantization is performed to obtain I i Corresponding quantized samples , constitute a quantized sample sequence ;

[0053] S4, construct a defense network M rec , used for input Output defense sample sequence ;

[0054] Therefore, the defense network M rec Based on the Unet network, when i=1 Defense samples are output via the Unet network. When i≥2, Input the Unet network and fuse the quantized samples from the previous frame. Post-output defense sample ;

[0055] S5, Obtain a tracker M based on a Siamese network. F The M F The feature extraction network has J layers, and each layer outputs a feature map.

[0056] The M F For input and And for M F Output Feature map of the corresponding j-th layer feature extraction network Category tags and regression labels ,right M F Output Feature map of the corresponding j-th layer feature extraction network Predicted classification labels and predictive regression labels , 1≤j≤J;

[0057] S6, M rec and M F Assemble the overall network and construct the overall loss function L. Tri Including feature loss L fea Defense sample classification loss L cls and defensive sample regression loss L reg ;

[0058] S7, preset iteration rounds, use and Train the overall network, and freeze M during training. F Network parameters to minimize L Tri Update M rec The network parameters are adjusted until the required number of iterations is reached, and the trained M is then... rec As a defense reconstruction model ;

[0059] S8: Obtain the video frame sequence to be identified, generate the corresponding quantized sample sequence according to steps S2 and S3, and then process it through the defense reconstruction model. Generate defense sample sequences for tracker tracking.

[0060] 0026. In this embodiment: S3 is used for... Quantization is performed to obtain quantized samples. Specifically;

[0061] Using the bit plane layering method Divided into the first bit plane b1 to the eighth bit plane b8; preset quantization factor k, according to the formula Generate quantized samples In the formula, 1≤k<8, and j is the j-th bit plane. The index. For example, when k=5, .

[0062] 0027. Step S4 includes steps S41 to S42;

[0063] S41, Obtain a Unet network, including shrinking and expanding paths, and quantize the samples. The shrinking path sequentially generates the first downsampled feature to the third downsampled feature. ~ The extended path sequentially generates the third upsampled feature to the first upsampled feature. ~ ,and and , and , and , jump connection;

[0064] S42, for When i=1, the defense sample is directly output through the Unet network. ;

[0065] When i = 2 to n, improve the skip connection method of the Unet network, let and , and , and Skip connections to generate defense samples ,in, , , The clean sample I of the (i-1)th frame i-1 Corresponding quantized samples The first downsampling feature, the second downsampling feature, and the third downsampling feature.

[0066] 0028. In step S6, the feature loss L fea Total loss function L Tri It is obtained from the following formula;

[0067] ,

[0068] ,

[0069] In the formula, α, β, and γ are respectively L fea L cls and L reg The hyperparameter balancing factors. In this embodiment, α=0.4, β=0.3, and γ=0.3.

[0070] 0029. The tracker M F This includes SiamRPN, SiamRPN++, TransT, and STARK.

[0071] 0030. In step S6, the defensive sample classification loss L cls The cross-entropy loss function is used, and the defensive sample regression loss L reg Smooth L1 loss is used.

[0072] 0031. The training method for one iteration in step S7 includes:

[0073] S71, Enter M rec generate ;

[0074] S72, , Enter M F Calculate the total loss function L Tri ;

[0075] S73, using ADAM optimizer via L Tri Update M rec Network parameters.

[0076] 0032. Regarding the generation of defense samples from quantized samples, see Figures 2 and 3. Figure 2 is a diagram of the Unet network structure in the prior art. The left side of this network represents the shrinkage path and the quantized sample... Output in sequence ~ The right side shows the extended path, which is output sequentially. ~ and in and , and , and Skip connections are established between operations. In Figure 2, Conv represents the convolution operation, Up-Conv represents the deconvolution upsampling operation, 3×3, 2×2, and 1×1 represent the kernel sizes of the corresponding convolution operations, and ReLU is the activation function. The defense samples are obtained in this way, which corresponds to the processing flow in the first row of Figure 3. Figure 3 is a flowchart of the defense network of the present invention. It is based on Figure 2, only changing the way of hopping connections. For example, for... No longer according to existing technology and , and , and Instead of skipping connections, it makes... and , and , and Skip connections to generate defense samples .

[0077] 0033. The tracker based on Siamese networks described in this invention has a workflow mainly divided into three parts: feature extraction, feature fusion, and prediction. The Siamese network described in this invention refers to the feature extraction part using two identical feature extraction networks.

[0078] 0034. Regarding CNN trackers, taking the SiamRPN tracker as an example, SiamRPN includes a Siamese network and an RPN network. The Siamese network consists of two CNN networks with identical structural parameters. Template frames and search region frames are input into the Siamese network, where features are extracted by a CNN network. These features are then input into the RPN network, which includes classification and regression branches, to generate proposal boxes and predict the target's location and confidence. The CNN network performs feature extraction, while the RPN network performs feature fusion and predicts the target's location and confidence.

[0079] 0035. Regarding CNN trackers, taking the SiamRPN++ tracker as an example, feature extraction in the SiamRPN++ tracker still uses a Siamese network, but two ResNet50 networks with identical structural parameters are used to replace the CNN network in SiamRPN. ResNet50 performs multi-layer feature extraction, with each layer corresponding to a feature map, such as the feature map of the j-th layer feature extraction network described in this invention. .

[0080] 0036. Regarding trackers combining CNN and Transformer, taking the TransT tracker as an example, it includes three modules: a Siamese network, a Transformer-based feature fusion network, and a prediction head. The Siamese network uses two ResNet50s as feature extraction networks. After feature extraction, 1×1 convolutional layers are used to reshape these features before inputting them into the Transformer-based feature fusion network. Each layer in this feature fusion network has a Self-Context Enhancement (ECA) module and a Cross-Feature Enhancement (CFA) module to enhance self-attention and cross-attention. Finally, the fused features are input into the prediction head, which uses simple classification and regression branches to locate the target and find the predicted bounding box position. In the CNN backbone network, ResNet50 performs multi-layer feature extraction, with each layer corresponding to a feature map, for example, the feature map of the j-th layer feature extraction network. .

[0081] 0037. Regarding trackers that combine CNN and Transformer, taking the STARK tracker as an example, feature extraction is still based on ResNet for multi-layer feature extraction.

[0082] 0038. The tracker in Figure 1 uses the TransT tracker structure as an example, including a Siamese network, a feature fusion network, and a prediction head. The Siamese network consists of two ResNet50 networks as feature extraction networks, with each ResNet50 performing J layers of feature extraction.

[0083] 0039. The defense reconstruction model trained by the method of this invention It can be plug-and-play applied to other trackers and has good transferability. For example, to identify a sequence of video frames, it is only necessary to generate the corresponding quantized sample sequence according to steps S2 and S3, and then process it through the defense reconstruction model. A defense sample sequence is generated, and then the defense sample sequence is input into the tracker, which can then perform target tracking normally according to existing technology.

[0084] 0040. Example 2: To test the defense method proposed in this invention, we conducted tests on three benchmark datasets: OTB100, VOT2018, and LaSOT. For the hyperparameter settings, we used a quantization factor k=5, α=0.4, β=0.3, and γ=0.3, and employed the classic white-box attack CSA method (template and search region attack) and the black-box attack IoU method (search region attack) to test the defense performance of this invention. The experimental results are shown in Table 1:

[0085] Table 1: Defense Results of the Invention

[0086]

[0087] In Table 1, the metrics are: Success (success rate), Precision (precision rate), Accuracy (accuracy rate), Robustness (robustness), Norm. Precision (normalized precision), and EAO (Expected Average Overlap) (expected average coverage).

[0088] 0041. Experiments have shown that the BPDN defense network proposed in this invention can significantly improve the robustness of the tracker, maintain good generalization against various adversarial attack methods such as black-box and white-box attacks, perform well on clean samples, and the defense network not only has low training cost and high efficiency, but can also be plugged and used to other trackers, with good transferability.

[0089] 0042. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A bit-plane based auxiliary single-target tracking counter-defense reconstruction method, characterized in that: Comprising the following steps; S1, acquiring a sequence of video frames consisting of clean samples where the ith clean sample is I i , which is labeled ， 、 respectively, I i classification and regression tags, n is the total number of clean samples; S2, in I i adding perturbations I is obtained i corresponding adversarial sample , constituting an adversarial sample sequence ； S3, the bit plane layering method is applied to Quantization is performed to obtain I i Corresponding quantized samples , constituting a quantized sample sequence ； S4, constructing a defense network M rec , for input , outputting a defense sample sequence ； So the defense network M rec Based on the Unet network, when i = 1, defense samples outputted by the unet network when i > 2, inputting the unet network and fusing the previous frame quantized samples Post-output defense samples ； S5, obtaining a tracker M based on a twin network F , the M F characteristic extraction network of the M has J layers, and each layer outputs a feature map; The M F For input and , and to , M F output feature map of the corresponding jth layer feature extraction network , classification label and regression labels , to , M F output feature map of the corresponding jth layer feature extraction network , predicted classification label and predicting regression labels , 1≤j≤J; S6, M rec and M F composing a total network, constructing a total loss function L Tri , including feature loss L fea , defense sample classification loss L cls and defense sample regression loss L reg ; S7, preset iteration round, use and Train the overall network, and freeze the network parameters of M F during training to minimize L Tri Update the network parameters of M rec until a number of iteration rounds is reached, and use the trained M rec as a defense reconstruction model ； S8, acquire a sequence of video frames to be identified, generate a corresponding sequence of quantized samples according to steps S2 and S3, and then pass the sequence of quantized samples through the defense reconstruction model Generating a defense sample sequence for the tracker to track.

2. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 1, characterized in that: S3 is quantized to obtain quantized samples Specifically;​ by a bit-plane layering method are divided into a first bit plane b1 to an eighth bit plane b8; a preset quantization factor k, according to the formula Generating quantification samples where 1≤k<8, j is the jth bit plane Index of the.

3. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 2, characterized in that: k=5。 4. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 1, characterized in that: Step S4 comprises steps S41-S42; S41, acquire a Unet network, including a shrink path and an expansion path, to the quantized sample , the contraction path generates the first down-sampled feature to the third down-sampled feature in turn ~ , the extended path generates third up-sampling features in sequence from the first up-sampling features ~ , and With 、 With 、 With Skip connection; S42, to when i = 1, the defense sample is directly outputted through the Unet network ； When i = 2~n, the improved Unet network jump connection mode is changed to let With 、 With 、 With Jumping connections, generating defensive samples wherein, 、 、 respectively the i-1th frame clean sample I i-1 corresponding quantized sample First down-sampling feature, second down-sampling feature, third down-sampling feature of the.

5. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 1, characterized in that: In S6, the feature loss L fea , the total loss function L Tri is obtained according to the following formula; , , wherein a, β, γ are the hyperparameter balancing factors for L fea , L cls , and L reg , respectively.

6. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 5, characterized in that: Alpha=0.4, beta=0.3, gamma=0.

3.

7. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 1, characterized in that: The tracker M F includes SiamRPN, SiamRPN++, TransT, and STARK.

8. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 1, characterized in that: In S6, the defense sample classification loss L cls The defense sample regression loss L is adopted using a cross-entropy loss function reg Smooth L1 loss is adopted.

9. The auxiliary single-target tracking counter-defense reconstruction method based on bit planes according to claim 1, characterized in that: The training method of one iteration in S7 comprises; S71, to Input M rec Generate ； S72, to 、 Input M F Compute total loss function L Tri ; S73, use the ADAM optimizer to optimize the network parameters by L Tri update M rec network parameters.

Images