Cross-modal person re-identification method and system based on wavelet transform

Abstract

The present invention relates to the technical field of person re-identification. Disclosed are a cross-modal person re-identification method and system based on wavelet transform. The method comprises: constructing a two-stream ResNet-50 backbone network; by means of an ICB module, fusing shallow features of the two-stream ResNet-50 backbone network, so as to obtain fused features; by means of a WEB branch module, performing feature extraction on the fused features, so as to obtain modality-shared features of high-frequency and low-frequency portions; and by means of a dual-branch center-guided loss, optimizing the modality-shared features output by the WEB branch module and the fused features output by the ICB module, so as to obtain a final identification result. In the present invention, an information compensation module is combined with wavelet transform to aggregate shallow network features in different phases, and thus valuable information lost in network feature extraction is compensated for, thereby improving the quality of a final feature vector; and a wavelet enhancement module is provided, and a dual-branch center-guided loss is used to guide a network to mine modality-invariant information in wavelet subgraphs, thereby improving the model performance.

Description

A cross-modal person re-identification method and system based on wavelet transform Technical Field

[0001] This invention relates to the field of pedestrian re-identification technology, and more specifically to a cross-modal pedestrian re-identification method and system based on wavelet transform. Background Technology

[0002] Person re-identification (ReID) aims to retrieve a specific person from multiple disjoint camera images, playing an increasingly important role in intelligent video surveillance. Existing ReID methods mainly focus on information retrieval between RGB images. However, with increasing public safety awareness and the widespread use of infrared cameras, more and more surveillance is conducted at night or in low-light conditions, and models designed for visible modalities often face low accuracy in nighttime or low-light environments. Recent research has addressed an important but challenging ReID subtask: visible-infrared ReID (VI-ReID), which aims to match the same person across modalities (visible to infrared, infrared to visible).

[0003] The main challenge of VI-ReID is extracting modality-independent identity features to combat the significant modal differences between visible and infrared modalities. Based on current work, two main approaches are commonly used to mitigate these differences. One is image-level methods, which convert images from two different modalities to the same modality to maintain style consistency. However, introducing additional noise during image conversion can affect the extraction of discriminative features. The other is feature-level methods, which utilize neural networks to extract identity features from different modalities and map them to a shared space for alignment. These methods effectively handle modality differences, but they still cannot avoid losing valuable information during feature extraction, including modality-independent features that can be mined and utilized.

[0004] In recent years, wavelet transform feature learning has demonstrated powerful advantages in various tasks such as image classification, medical image segmentation, and image denoising. Wavelet transform can decompose feature maps into high-frequency and low-frequency components. The low-frequency component contains less noise and detail, while the high-frequency component contains clearer object boundary information, such as contours. Applying wavelet transform to feature maps helps models mine shared features between modalities from different perspectives. Furthermore, wavelet transform (DWT) and inverse wavelet transform (IDWT) of feature maps are lossless, without any additional information loss. Traditional feature extraction-based methods often fail to fully extract invariant information within modalities and are easily affected by noise features. Wavelet transform is an effective time-frequency analysis method. Considering the reversibility and ability to preserve all information, wavelet transform has been used to improve performance in various vision tasks. Although wavelet transform has been used in image classification, medical image segmentation, and image denoising, no related research has been seen in the VI-Reid field to date.

[0005] Therefore, how to propose a cross-modal person re-identification method and system based on wavelet transform, combine wavelet transform with attention mechanism, and use wavelet transform to decompose feature map, mine modality-invariant features in the decomposed low-frequency and high-frequency images, reduce modality differences, and realize VI-ReID is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a cross-modal person re-identification method and system based on wavelet transform. It combines wavelet transform with an attention mechanism and uses wavelet transform to decompose the feature map. Modality-invariant features are mined from the decomposed low-frequency and high-frequency images to reduce modality differences and achieve VI-ReID. To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A cross-modal person re-identification method based on wavelet transform includes:

[0008] The test images of VI-ReID are acquired and used to construct a two-stream ResNet-50 backbone network;

[0009] The fused features are obtained by fusing the shallow features of the two-stream ResNet-50 backbone network using the ICB module;

[0010] The WEB branch module extracts features from the fused features to obtain modal sharing features of high and low frequency components;

[0011] The modality-sharing features output by the WEB branch module and the fusion features output by the ICB module are optimized by using a dual-branch-center guided loss to obtain the final recognition result.

[0012] The final identification results are then visually verified.

[0013] Optionally, the ICB module is used to fuse shallow features of the two-stream ResNet-50 backbone network. The ICB module integrates self-attention learning with wavelet transform, uses wavelet transform for reversible downsampling to reduce the spatial dimension of deep features, performs self-attention learning with shallow features, and concatenates the result with deep features that have undergone inverse wavelet transform to obtain fused features.

[0014] Optionally, the wavelet transform includes:

[0015] The original image is decomposed into LF component, horizontal HF component, vertical HF component and diagonal HF component using wavelet transform, resulting in LL, HL, LH and HH. LL represents the low frequency part, and LH, HL and HH represent the high frequency part. The four decomposed components are defined as L and H parts.

[0016] Optionally, the step of concatenating the result with the deep features obtained by inverse wavelet transform to obtain the fused features includes: adding four ICB modules to the backbone network ResNet; the first three ICB modules perform feature fusion before and after each ResNetBlock; and the fourth ICB module performs feature fusion between the outputs of the first and third ICB modules.

[0017] Optionally, the ICB module structure includes:

[0018] The features before and after each ResNetBlock are used as shallow and deep features respectively, and together they are used as input to the ICB module. Convolution and regularization are applied to the input to map it to a feature space of the same dimension. The number of channels of the deep features is reduced to obtain the reduced features.

[0019] The reduced feature is decomposed into n wavelet subbands using Haar discrete wavelet transform;

[0020] Concatenate the n wavelet subbands along the channel dimension to obtain the output of the Haar discrete wavelet transform.

[0021] The output of the Haar discrete wavelet transform is convolved with shallow features to capture features of local regions, and the attention output is calculated.

[0022] Design a DWT-Conv-IDWT structure, and concatenate the attention output with the output of the DWT-Conv-IDWT structure to obtain the output of the wavelet attention block;

[0023] The output of the wavelet attention block is convolved to recover the number of channels, and then added to the deep features of the initial input to obtain the final output.

[0024] Optionally, the step of extracting high- and low-frequency modal-shared features from the fused features using the WEB branch module includes:

[0025] The output features after ResNetblock are decomposed into n wavelet subbands by performing Haar discrete wavelet transform on them, and these n wavelet subbands are concat in the channel dimension to obtain the extended result.

[0026] The expanded results are input into BottleNeck convolutional blocks for subtle modality-consistent feature learning. Finally, an inverse wavelet transform is applied to the output of the BottleNeck block structure to restore the output feature map dimension to its original size.

[0027] Optionally, it also includes: concatenating the output features of the original ResNetblock with the output features of the WEB branch module and inputting them into the last ResNetblock to obtain n sets of features generated by the dual branches.

[0028] Optionally, the optimization of the modality-sharing features output by the WEB branch module and the fusion features output by the ICB module using the dual-branch-center guided loss to obtain the final recognition result includes:

[0029] Calculate the center of each of the n sets of features generated by the bi-branch;

[0030] The criteria for bi-branch center-guided loss are set, and the distance loss of the centers of n sets of features is defined by measuring the similarity of features through Euclidean distance.

[0031] The bi-branch center-guided loss is derived by taking the mean of the distance loss between the centers of the defined n sets of features.

[0032] Optionally, it also includes calculating the total loss as:

[0033] Where λ is a coefficient that controls the proportion of loss. For the loss of identity, The triple entropy loss is achieved by minimizing the three types of loss. The sum of these factors together optimizes the network.

[0034] Optionally, a cross-modal person re-identification system based on wavelet transform includes:

[0035] Backbone network construction module: used to acquire the VI-ReID test image and use it to construct a two-stream ResNet-50 backbone network;

[0036] Feature fusion module: Used to fuse shallow features from two streams of the ResNet-50 backbone network using the ICB module to obtain fused features;

[0037] Feature extraction module: used to extract features from the fused features through the WEB branch module to obtain modality-shared features of high and low frequency components;

[0038] Optimization module: Used to optimize the modality-sharing features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss, to obtain the final recognition result;

[0039] Verification module: Used to perform visual verification of the final recognition results.

[0040] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a cross-modal pedestrian re-identification method and system based on wavelet transform, which has the following beneficial effects:

[0041] This invention proposes a cross-modal person re-identification method based on wavelet transform, comprising: acquiring a VI-ReID image to be tested and constructing a two-stream ResNet-50 backbone network; fusing shallow features of the two-stream ResNet-50 backbone network through an ICB module to obtain fused features; extracting features from the fused features through a WEB branch module to obtain modal-shared features of high and low frequency components; optimizing the modal-shared features output by the WEB branch module and the fused features output by the ICB module through a dual-branch center guided loss to obtain the final recognition result; and visually verifying the final recognition result. This invention proposes a novel feature mining method based on wavelet transform, aiming to mine latent information in wavelet subgraphs for cross-modal person re-identification tasks. An information compensation module is proposed, which combines wavelet transform with shallow network features at different stages to compensate for valuable information lost during feature extraction, thus improving the quality of the final feature vector. A wavelet enhancement module is also proposed, using a dual-branch center-guided loss to guide the network to fully mine modality-invariant information in the wavelet subgraphs, improving model performance. Extensive experiments on three challenging VI-ReID datasets demonstrate that the proposed method outperforms other state-of-the-art methods. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 is a schematic diagram of the process of a cross-modal pedestrian re-identification method based on wavelet transform provided by the present invention.

[0044] Figure 2 is a diagram of the wavelet transform feature mining network (WFM) framework provided by the present invention.

[0045] Figure 3(a) is a schematic diagram of the two-dimensional image application DWT and IDWT provided by the present invention.

[0046] Figure 3(b) is a schematic diagram of the specific DWT process provided by the present invention.

[0047] Figure 3(c) is a schematic diagram of the receptive field of DWT-Conv-IDWT provided by the present invention.

[0048] Figure 4 is a structural framework diagram of the ICB module provided by the present invention.

[0049] Figure 5(a) is a schematic diagram of the WEB module provided by the present invention.

[0050] Figure 5(b) is a central schematic diagram of the calculation of the four sets of features provided by the present invention.

[0051] Figure 5(c) shows the invention provided by the present invention. Schematic diagram.

[0052] Figure 6(a) is a schematic diagram showing the effect of hyperparameter λ on model performance on the SYSU-MM01 dataset provided by this invention.

[0053] Figure 6(b) is a schematic diagram showing the effect of hyperparameter ρ on model performance on the SYSU-MM01 dataset provided by this invention.

[0054] Figure 7(a) shows the rank-10 search results of the baseline provided by this invention on the SYSU-MM01 dataset.

[0055] Figure 7(b) shows the rank-10 search results of WFM on the SYSU-MM01 dataset provided by this invention.

[0056] Figure 8 is a thermal visualization of the various modules provided by the present invention.

[0057] Figure 9 shows the intra-class and inter-class distances of the cross-modal features provided by the present invention.

[0058] Figure 10 is a distribution diagram of the feature embeddings provided by the present invention in a two-dimensional feature space. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] This invention discloses a cross-modal person re-identification method based on wavelet transform, as shown in Figure 1, including:

[0061] The test images of VI-ReID are acquired and used to construct a two-stream ResNet-50 backbone network;

[0062] The fused features are obtained by fusing the shallow features of the two-stream ResNet-50 backbone network using the ICB module;

[0063] The WEB branch module extracts features from the fused features to obtain modal sharing features of high and low frequency components;

[0064] The modality-sharing features output by the WEB branch module and the fusion features output by the ICB module are optimized by using a dual-branch-center guided loss to obtain the final recognition result.

[0065] The final identification results are then visually verified.

[0066] In a specific implementation, a cross-modal person re-identification method based on wavelet transform, as shown in Figure 2, applies wavelet transform to the cross-modal person re-identification task and proposes a wavelet transform feature mining network (WFM). This network can learn discriminative information from the feature maps of wavelet decomposition and mine various subtle modal shared features contained therein, specifically including:

[0067] The network employs an Information Compensation Module (ICB), a Wavelet Transform Enhancement Module (WEB), and a dual-branch center-guided loss. The ICB module compensates for lost valuable information during feature extraction by fusing current features with those from the previous stage. The WEB module, through a new branch, allows the network to further mine mode-invariant features. To guide the module in extracting key information different from the other branch, a dual-branch guided loss is proposed to improve the model's feature learning ability. The Information Compensation Module (ICB) and Wavelet Transform Enhancement Module (WEB) are used to reduce the modal differences between visible light and infrared images in the frequency domain. The dual-branch center-guided loss L... dbcThe proposed WFM guides the WEB module to discover modality-shared features on the feature map reconstructed by wavelet transform. Visualization results demonstrate that the model pays stronger attention to modality-invariant features and reduces the influence of interfering information. Extensive experiments show that the proposed WFM achieves excellent performance on three challenging cross-modal datasets.

[0068] Specifically, the Wavelet Transform-Based Feature Mining Network (WFM) uses a two-stream ResNet-50 network as its backbone. First, an ICB module is designed to fuse shallow features. The ICB module integrates self-attention learning with wavelet transform, using wavelet transform for reversible downsampling to reduce the spatial dimension of deep features without information loss. Self-attention learning is then performed with the shallow features, and the result is concatenated with the deep features after inverse wavelet transform to obtain the fused features. Furthermore, a WEB branch module is designed. This branch extracts features from the decomposed wavelet subgraphs to mine modality-shared features in the high- and low-frequency components, reducing modality differences. Finally, a dual-branch center-guided loss L is proposed. dbc This guides the model to learn additional modal-shared features from the WEB branch to reduce intra- and inter-class differences.

[0069] In a specific implementation, wavelet transform specifically includes:

[0070] Two-dimensional images are essentially two-dimensional discrete non-stationary signals, containing different frequency ranges and spatial location information. Wavelet transform can effectively preserve this information while decomposing it. As shown in Figure 3(a), wavelet transform is used to decompose the original image into LF, horizontal HF, vertical HF, and diagonal HF components (i.e., LL, HL, LH, and HH, where LL represents the low-frequency component, and LH, HL, and HH represent the high-frequency component). The four decomposed components are defined as two parts: L and H.

[0071] The L component has less noise and less detail. In contrast, the H component has more noise but clearer object boundaries, which helps the model focus more on details. Wavelet transform lossless decomposition helps the model extract modal-shared features across different components, improving model performance.

[0072] In specific implementations, as network depth increases, the extracted features become increasingly refined, and some shallow spatial information is discarded. Reasonably fusing shallow features containing rich spatial information with deep features can effectively improve model performance. To address this, an Information Compensation Module (ICB) based on wavelet transform is designed to reuse the spatial information lost in the shallow layers of the network, as shown in Figure 4. DWT and IDWT represent wavelet transform and inverse wavelet transform, respectively. The ICB module specifically includes:

[0073] 1) Use the features before and after each residual block as shallow features respectively. and deep features Together they serve as input to the ICB module, for the input x l and x h Apply convolution and regularization to map them to a feature space of the same dimension.

[0074] 2) Further x h The number of channels is reduced to 1 / 4 of the original number, denoted as X;

[0075] 3) Input The feature map X is decomposed into four wavelet subbands using Haar Discrete Wavelet Transform (DWT). The DWT decomposition process is shown in Figure 3(b). A low-pass filter is applied along the input feature map X.

[0076] and high-pass filter Encode X into two subbands X L and X H Next, along the learned subband X... L and X H The columns use the same low-pass filter f L and high-pass filter f H Thus, all four wavelet subbands are obtained: X LL This refers to low-frequency components that reflect the basic structure of an object at a coarse-grained level. X LH X HL and X HH This represents a high-frequency component that preserves object texture details at a fine-grained level. Finally, the four wavelet subbands are concatenated along the channel dimension to obtain the DWT output.

[0077] 4) Output of DWT and shallow features X l Convolution is used to capture features in local regions, enhancing the model's ability to perceive local information in the feature map: ψ Q =Conv q (x l );

[0078] Where G represents a 3×3 convolution, Conv q Conv k Conv v It's a 1×1 convolution. Let ψ Q , ψ k , ψ v Q, K, and V are attention mechanisms.

[0079] 5) For ψ Q , ψ k After matrix multiplication, the attention matrix A is obtained by applying the softmax activation function: A = Softmax(ψ) Q ×ψ K );

[0080] Compare attention matrix A with Conv v Multiplying them together yields the attention output.

[0081] 6) Design the DWT-Conv-IDWT structure, as shown in Figure 3(c). This structure has a larger receptive field compared to ordinary convolution, thereby increasing the network's context-local awareness capability. Concatenate the attention output with the DWT-Conv-IDWT output to obtain the wavelet attention block output:

[0082] 7) Output X of the wavelet attention block w Perform a 1×1 convolution to recover the number of channels, and add it to the initial deep features to obtain the final output. out =F conv (x w )+x h ;

[0083] This embodiment adds four ICB modules to the ResNet backbone network. The first three perform feature fusion before and after each ResNetBlock. The fourth performs feature fusion between the outputs of the first and third ICB modules. The aim is to fully utilize wavelet transform to enhance local perception capabilities to fuse important spatial information from shallow networks, enabling the network to better focus on modality-shared features.

[0084] In a specific implementation, the wavelet enhancement module specifically includes:

[0085] As shown in Figure 5(a), a new branch is added after the third residual block of ResNet, aiming to improve model performance by mining mode-invariant features from the feature map after wavelet transform. Specifically, for the output feature X after the third block... in Perform a Harr Discrete Wavelet Transform (DWT) to decompose the wavelet into four wavelet subbands, and then concatenate these four wavelet subbands along the channel dimension, as follows:

[0086] Among them, X LL ,X LH ,X HL ,X HH These are the four wavelet subbands obtained after DWT decomposition, and Concat represents the splicing operation on the channel.

[0087] Will The input is fed into BottleNeck convolutional blocks for subtle modality-consistent feature learning;

[0088] Applying an inverse wavelet transform to the output of the BottleNeck convolutional block structure restores the output feature map dimension to its original size. This process is described as follows:

[0089] The output features of the original block3 are concatenated with the output features of the WEB module and input into the last block, ultimately resulting in four sets of features generated by the dual branches:

[0090] In a specific implementation, the steps for determining the dual-branch center guidance loss are as follows:

[0091] Although the decomposed wavelet subgraphs contain clearer details, neural networks struggle to learn this information autonomously, which contradicts the original design intent of the web module. To address this, a dual-branch center-guided loss mechanism is employed. To enhance the web's ability to mine subtle modal shared features. As shown in Figure 5(b), for pedestrians with the same ID, first calculate... The center of the 4 sets of features:

[0092] Where K represents the number of pedestrians with the same ID in each group, and c represents the feature centers of all pedestrians with the same ID in each group. V and T represent the visible light and infrared modes, respectively, and o and w represent the original features and the wavelet features processed by the WEB module, respectively.

[0093] Design The three principles are as follows: the visible light modal features and the infrared modal features should be as similar as possible, in order to encourage the network to focus on modality-shared features and reduce modality differences; the visible light / infrared modal features generated by the WEB module should be as different as possible from the original visible light / infrared modal features, in order to encourage the two branches of the network to produce complementary features and enrich the feature representation of the network; and the different ID features should be as different as possible, in order to encourage the network to retain identity discrimination information.

[0094] Euclidean distance is used to measure the similarity of features. Definition The distance loss is as follows:

[0095] Where D(·,·) represents the Euclidean distance between two features, + ensures the loss value is non-negative, and the loss is zero when the calculated value is less than zero. ρ represents the interval, which controls the distance between positive and negative samples.

[0096] As shown in Figure 5(c), Bridging the gap between the original visible light modal features and infrared modal features helps reduce modal differences between VIS and IR images. Increasing the distance between the visible light modal features generated by the WEB module and the original visible modal features helps the model to mine different cross-modal shared features in the WEB module. Both methods ensure that the intra-class distance is less than the inter-class distance by increasing the distance between different ID features.

[0097] Similarly, The distance loss between them is as follows:

[0098] Defined as:

[0099] In a specific implementation, the calculation of the total loss is also included:

[0100] Combined with identity loss and triple entropy loss By minimizing these three losses The sum of these factors, used to jointly optimize the network, is expressed as: The total loss is:

[0101] Wherein, λ is a coefficient that controls the proportion of loss.

[0102] In a specific implementation, a cross-modal person re-identification system based on wavelet transform includes:

[0103] Backbone network construction module: used to acquire the VI-ReID test image and use it to construct a two-stream ResNet-50 backbone network;

[0104] Feature fusion module: Used to fuse shallow features from two streams of the ResNet-50 backbone network using the ICB module to obtain fused features;

[0105] Feature extraction module: used to extract features from the fused features through the WEB branch module to obtain modality-shared features of high and low frequency components;

[0106] Optimization module: Used to optimize the modality-sharing features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss, to obtain the final recognition result;

[0107] Verification module: Used to perform visual verification of the final recognition results.

[0108] In a specific embodiment, the visualization verification of the final identification result specifically includes:

[0109] S1: Dataset and Validation Metrics

[0110] (1) Dataset

[0111] SYSU-MM01 is a large dataset containing 491 pedestrians from 4 visible light cameras and 2 infrared cameras. The training set contains 19,659 visible images and 12,792 infrared images of 395 people, while the test set contains 96 people. There are two test modes: full search mode and indoor search mode. For full search mode, all images are used. For indoor search mode, only indoor images from the first, second, third, and sixth cameras are used.

[0112] RegDB is a small dataset containing images taken by a visible light camera and an infrared camera. It contains 8240 images for 412 identities, with each image consisting of 10 visible images and 10 thermal images. The dataset is randomly split into two parts: images for 206 identities are used for training, and images for the remaining 206 identities are used for testing.

[0113] LLCM was captured in low-light conditions by nine visible light cameras and nine infrared cameras, containing 46,767 images of 1,064 identities. The training set contains 16,946 VIS images and 13,975 IR images of 713 identities, while the test set contains 8,680 VIS images and 7,166 IR images of 351 identities.

[0114] (2) Evaluation indicators

[0115] Cumulative Matching (CMC) and Mean Average Precision (mAP) were used as the main evaluation metrics. CMC reflects the probability that the first n results in the retrieved image are correct, while mAP reflects the average retrieval precision for all identities.

[0116] S2: Implementation Details: The AGW dataset was used as the baseline, and weights pre-trained on ImageNet were used for initialization. During training, all input images were resized to 3×384×192 and augmented using horizontal flipping and random erasure. For each mini-batch, eight identities were randomly selected, with four VIS images and four IR images chosen for each identity. A warm-up strategy was used to update the learning rate from 0.01 to 0.1 in the first 10 training epochs. At epochs 20, 60, and 120, the learning rate was set to 0.01, 0.001, and 0.0001, respectively. Stochastic Gradient Descent (SGD) was used as the optimizer, with a momentum parameter set to 0.9. The total number of training iterations was set to 150, and all experiments were implemented using PyTorch on a single NVIDIA RTX 3090 GPU.

[0117] S3: Performance Comparison of Different Models

[0118] The WFM model was compared with state-of-the-art VI-ReID methods on the public VI-ReID datasets SYSU-MM01, RegDB, and LLCM. The experimental results are shown in Tables 1, 2, and 3.

[0119] SYSU-MM01: As shown in Table 1, the proposed WFM outperforms other state-of-the-art methods, achieving a Rank-1 of 78.02% and an mAP of 74.16% in the all-search mode, and a Rank-1 of 87.68% and an mAP of 87.62% in the indoor search mode. The proposed method shows better results in both Rank-1 and mAP in both global and indoor search modes. Furthermore, compared to other methods based on two-stream networks [SPOT, CAJ, DEEN], the proposed method also exhibits substantial performance advantages, with Rank-1 and mAP exceeding DEEN by 3.32% and 2.36% respectively in the all-search mode, and by 7.38% and 4.32% respectively in the indoor search mode. These results also demonstrate that the proposed method performs better in indoor environments. Among them, SPOT stands for structure-aware positional transformer (SPOT) network; CAJ stands for channel augmented joint (CAJ) network; and DEEN stands for Diverse Embedding Expansion Network.

[0120] Table 1 Performance comparison of various models on the SYSU-MM01 dataset

[0121] RegDB: As shown in Table 2, the proposed WFM achieves the best performance among state-of-the-art methods. The rank-1 and mAP for the visible-to-infrared mode are 92.23% and 86.16%, respectively, while the mAP for the infrared-to-visible mode is 91.94% and 85.70%, respectively. The results demonstrate that this method exhibits excellent performance in both visible-to-infrared and infrared-to-visible modes. In the visible-to-infrared mode, the rank-1 and mAP are 1.13% and 1.06% higher than DEEN, respectively, and in the infrared-to-visible mode, the rank-1 and mAP are 2.44% and 2.30% higher than DEEN, respectively. This reflects the strong robustness of the proposed method to different datasets and query patterns.

[0122] Table 2 Performance comparison of various models on the RegDB dataset

[0123] LLCM: As shown in Table 3, the proposed WFM also achieves excellent performance on the LLCM dataset in low-light complex scenes. The Rank-1 and mAP from visible to infrared modes are 63.61% and 66.12%, respectively, higher than DEEN by 1.11% and 0.32%. The mAP from infrared to visible modes are 55.74% and 63.10%, respectively, higher than DEEN by 0.84% ​​and 0.2%. This indicates that the WFM model also exhibits strong robustness in low-light complex multimodal scenes.

[0124] Table 3 Performance comparison of various models on the LLCM dataset

[0125] S4: Ablation Experiments: Ablation experiments were conducted to evaluate the contribution of each component in the proposed WFM. For fair comparison, all experiments were performed on SYSU-MM01 using both full search and indoor search modes, with AGW used as the baseline for all experiments and the same data augmentation employed.

[0126] (1) Validity of each module

[0127] The analysis focuses on the key components of WFM, including the Information Compensation Module (ICB), the Wavelet Enhancement Module (WEB), and the dual-branch center-guided loss. As shown in Table 4, each module improved the model's performance. The ICB module, by fusing shallow features at different stages, reused the shallow information lost as the network deepened, improving Rank-1 and mAP by 5.59% and 3.65% respectively across all search modes from the first to the third row. It's worth noting that adding the WEB module alone slightly improved performance, but the results were not ideal. Rank-1 and mAP improved by only 3.21% and 0.18% respectively across all search modes from the first to the second row. This is because the model doesn't actively mine the modality-invariant information of the feature maps reconstructed by wavelet transform; a loss function is needed to guide the network to mine modality-shared features. (The text abruptly ends here, likely due to an incomplete sentence or missing information.) Subsequently, performance was further improved. In full search mode, from the second to the fourth row, Rank-1 and mAP increased by an additional 0.86% and 0.74%, respectively; from the fifth to the sixth row, Rank-1 and mAP increased by an additional 1.53% and 2.13%, respectively, indicating that... This greatly enhances the model's ability to discover modal-shared features.

[0128] Table 4. Impact of each WFM module on model performance

[0129] Table 5 Comparison with Non-Local (NL) Module

[0130] Wherein, ICB* represents the ICB module without wavelet transform.

[0131] (2) Comparison with Non-Local (NL) modules

[0132] The Information Compensation Module (ICB) based on wavelet transform is similar in structure to the Non-Local module, aiming to fuse features from different stages. Comparative experiments were conducted between ICB and Non-Local, as shown in Table 5. In the all-search mode, the ICB module outperformed the Non-Local module in Rank-1 and mAP by 1.07% and 1.62%, respectively. In the indoor search mode, the ICB module outperformed the Non-Local module in Rank-1 and mAP by 1.65% and 0.96%, respectively. These results verify that the ICB module, incorporating wavelet transform, performs better than the Non-Local structure. Comparing rows 3 and 4 of Table 5, in the all-search mode, wavelet transform improved the model's Rank-1 by 1.52% and mAP by 1.84%, demonstrating that wavelet transform is essential and plays a crucial role in the ICB module.

[0133] (3) The influence of hyperparameters

[0134] The impact of the hyperparameters λ and ρ on network performance was evaluated and quantitatively compared, with results shown in Figures 6(a) and 6(b). It can be seen that the performance of WFM gradually improves with increasing λ and ρ values. However, when λ and ρ exceed a certain threshold, the performance of WFM decreases. Therefore, λ = 0.8 and ρ = 0.2 were set to achieve optimal performance.

[0135] S5: Visualization

[0136] (1) Rank visualization

[0137] To demonstrate the advantages of WFM, a full-search test mode method was used on the SYSU-MM01 dataset for comparison with the baseline. The top 10 images with the highest matching scores are shown in Figures 7(a) and 7(b). The WFM method effectively handled modal differences and correctly identified images of the same person (checkmarked below). In contrast, the baseline was insensitive to modality-invariant features and easily interfered with by non-critical information, such as the hand-in-pocket pose in row 4, resulting in a large number of incorrectly matched images (crossed below). In each retrieval case, the images in the first column are infrared queries, while the remaining images are visible images from the library. A checkmark under an image indicates that the image matches the query ID, and a cross under an image indicates that the image does not match the query ID.

[0138] (2) Heatmap visualization

[0139] To explore the impact of each module on the network, a heatmap of the network after adding each module is shown in Figure 8. (a) represents the original image, and (b) represents the heatmap visualization of each module. Square regions represent noise information, and elliptical regions represent modality-shared features. It can be seen that compared to the baseline, WFM effectively enhances the network's extraction of modality-shared features, represented by ellipses, such as facial information, texture information on clothing, and shoe information. Simultaneously, it reduces the impact of noise information, represented by boxes, such as table corners and blackboards. Columns (c) and (d) show that adding the WEB or ICB modules individually also enhances the model's extraction of key features, demonstrating the effectiveness of each WFM module.

[0140] (3) Visualization of intra-class and inter-class distances

[0141] Figure 9(a)-(e) shows the intra-class and inter-class distances for cross-modal features. Intra-class and inter-class distances are represented by dark gray and light gray, respectively. Figure 10(f)-(j) shows the distribution of feature embeddings in the two-dimensional feature space, where circles and triangles represent visible and infrared modes. A total of 15 individuals were selected from the test set. Samples of the same color belong to the same individual. Circles and triangles indicate images from the VIS and IR modes, respectively.

[0142] To evaluate the effectiveness of this method, the frequency distribution of intra-class and inter-class distances between VIS and IR features on SYSU-MM01 was analyzed. The distance D between two features can be expressed as:

[0143] D(A,B)=1-CosSim(A,B);

[0144] CosSim(·,·) represents cosine similarity. As shown in Figures 9(a)-(e), each ID is considered as a class, and 1000 classes are selected to form a histogram. δ1, δ2, δ3, δ4, and δ5 represent the intra-class and inter-class peak distances for the initial, baseline, WEB, ICB, and WFM classes, respectively. The intra-class distance reflects the degree of clustering of pedestrians of the same class in the feature space (the smaller the value, the higher the clustering), while the inter-class distance reflects the degree of separation of pedestrians of different classes in the feature space (the larger the value, the greater the separation between classes). The frequency reflects the number of features at that distance. It can be noted that δ1 < δ2 < δ3 < δ4 < δ5, indicating that each module strengthens the clustering of similar features, effectively reducing the modal differences between VIS and IR images of pedestrians of the same class, thus achieving better performance.

[0145] (4) t-SNE visualization

[0146] To more intuitively demonstrate the distribution of different features in the two-dimensional feature space, the t-SNE tool was used to visualize the feature distribution on the SYSU-MM01 dataset. As shown in Figures 10(f)-(j), 15 identities were randomly selected from the test set of the SYSU-MM01 dataset. Compared with the baseline feature distribution in Figure 10(f), as modules are added, the distance between features of different identities gradually increases, while the distance between features of the same identity gradually decreases. This indicates that WFM effectively distinguishes and aggregates features of pedestrians of the same type, reducing the VIS-IR modal differences.

[0147] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-modal person re-identification method based on wavelet transform, characterized in that, include: The test images of VI-ReID are acquired and used to construct a two-stream ResNet-50 backbone network; The fused features are obtained by fusing the shallow features of the two-stream ResNet-50 backbone network using the ICB module; The ICB module is used to fuse shallow features from a two-stream ResNet-50 backbone network. The ICB module integrates self-attention learning with wavelet transform, uses wavelet transform for reversible downsampling to reduce the spatial dimension of deep features, performs self-attention learning with shallow features, and concatenates the result with deep features that have undergone inverse wavelet transform to obtain fused features. The process of concatenating the results with the deep features after inverse wavelet transform to obtain the fused features includes: adding four ICB modules to the backbone network ResNet; the first three ICB modules perform feature fusion before and after each ResNetBloc k; and the fourth ICB module performs feature fusion between the outputs of the first and third ICB modules. The ICB module structure includes: The features before and after each ResNetBlock are used as shallow and deep features respectively, and together they are used as input to the ICB module. Convolution and regularization are applied to the input to map it to a feature space of the same dimension. The number of channels of the deep features is reduced to obtain the reduced features. The reduced feature is decomposed into n wavelet subbands using Haar discrete wavelet transform; Concatenate the n wavelet subbands along the channel dimension to obtain the output of the Haar discrete wavelet transform. The output of the Haar discrete wavelet transform is convolved with shallow features to capture features of local regions, and the attention output is calculated. Design a DWT-Conv-IDWT structure, and concatenate the attention output with the output of the DWT-Conv-IDWT structure to obtain the output of the wavelet attention block; The output of the wavelet attention block is convolved to recover the number of channels, and then added to the deep features of the initial input to obtain the final output; The WEB branch module extracts features from the fused features to obtain modal sharing features of high and low frequency components; The modality-sharing features output by the WEB branch module and the fusion features output by the ICB module are optimized by using a dual-branch-center guided loss to obtain the final recognition result. The final identification results are then visually verified.

2. The cross-modal person re-identification method based on wavelet transform according to claim 1, characterized in that, The wavelet transform includes: The original image is decomposed into LF component, horizontal HF component, vertical HF component and diagonal HF component using wavelet transform, resulting in LL, HL, LH and HH. LL represents the low frequency part, and LH, HL and HH represent the high frequency part. The four decomposed components are defined as L and H parts.

3. The cross-modal person re-identification method based on wavelet transform according to claim 1, characterized in that, The modal sharing features obtained by extracting high- and low-frequency components from the fused features through the WEB branch module include: The output features after ResNetblock are decomposed into n wavelet subbands by performing Haar discrete wavelet transform on them, and these n wavelet subbands are concat in the channel dimension to obtain the extended result. The expanded results are input into BottleNeck convolutional blocks for subtle modality-consistent feature learning. Finally, an inverse wavelet transform is applied to the output of the BottleNeck block structure to restore the output feature map dimension to its original size.

4. The cross-modal person re-identification method based on wavelet transform according to claim 3, characterized in that, Also includes: The output features of the original ResNetblock are concatenated with the output features of the WEB branch module and input into the last ResNetblock to obtain n sets of features generated by the dual branches.

5. The cross-modal person re-identification method based on wavelet transform according to claim 1, characterized in that, The optimization of the modality-sharing features output by the WEB branch module and the fusion features output by the ICB module using the dual-branch-center guided loss yields the following final recognition results: Calculate the center of each of the n sets of features generated by the bi-branch; The criteria for bi-branch center-guided loss are set, and the distance loss of the centers of n sets of features is defined by measuring the similarity of features through Euclidean distance. The bi-branch center-guided loss is derived by taking the mean of the distance loss between the centers of the defined n sets of features.

6. A cross-modal person re-identification method based on wavelet transform according to claim 5, characterized in that, This also includes calculating the total loss as follows: in, It is a coefficient that controls the proportion of loss. For the loss of identity, The triple entropy loss is achieved by minimizing the three types of loss. The sum of these factors, working together to optimize the network. The loss is guided by the dual-branch center.

7. A cross-modal person re-identification system based on wavelet transform, characterized in that, include: Backbone network construction module: used to acquire the VI-ReID test image and construct a two-stream ResNet-50 backbone network; Feature fusion module: Used to fuse shallow features from two streams of the ResNet-50 backbone network using the ICB module to obtain fused features; The ICB module is used to fuse shallow features from a two-stream ResNet-50 backbone network. The ICB module integrates self-attention learning with wavelet transform, uses wavelet transform for reversible downsampling to reduce the spatial dimension of deep features, performs self-attention learning with shallow features, and concatenates the result with deep features that have undergone inverse wavelet transform to obtain fused features. The process of concatenating the results with the deep features after inverse wavelet transform to obtain the fused features includes: adding four ICB modules to the backbone network ResNet; the first three ICB modules perform feature fusion before and after each ResNetBloc k; and the fourth ICB module performs feature fusion between the outputs of the first and third ICB modules. The ICB module structure includes: The features before and after each ResNetBlock are used as shallow and deep features respectively, and together they are used as input to the ICB module. Convolution and regularization are applied to the input to map it to a feature space of the same dimension. The number of channels of the deep features is reduced to obtain the reduced features. The reduced feature is decomposed into n wavelet subbands using Haar discrete wavelet transform; Concatenate the n wavelet subbands along the channel dimension to obtain the output of the Haar discrete wavelet transform. The output of the Haar discrete wavelet transform is convolved with shallow features to capture features of local regions, and the attention output is calculated. Design a DWT-Conv-IDWT structure, and concatenate the attention output with the output of the DWT-Conv-IDWT structure to obtain the output of the wavelet attention block; The output of the wavelet attention block is convolved to recover the number of channels, and then added to the deep features of the initial input to obtain the final output; Feature extraction module: used to extract features from the fused features through the WEB branch module to obtain modality-shared features of high and low frequency components; Optimization module: Used to optimize the modality-sharing features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss, to obtain the final recognition result; Verification module: Used to perform visual verification of the final recognition results.

Images